java - Gerrit fails to load "Starting Gerrit Code Review: FAILED" no errors in log

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I've installed Gerrit according to the Standalone Daemon Installation Guide with MySQL. All plugins match the Gerrit install 2.12.2.
When starting Gerrit with ~/gerrit_install/bin/gerrit.sh start the output is simply:
Starting Gerrit Code Review: FAILED
There are no errors in ~/gerrit_install/logs/error_log
If I modify gerrit.sh to #!/bin/sh -x and run in gerrit.sh start, I see:
# bin/gerrit.sh start
+ test 1 -gt 0
+ ACTION=start
+ shift
+ test 0 -gt 0
+ test -z
+ NO_START=0
+ test -z
+ START_STOP_DAEMON=1
+ test -f /etc/default/gerritcodereview
+ test -z
+ TMP=/tmp
+ TMPJ=/tmp/j4726
+ GERRIT_INSTALL_TRACE_FILE=etc/gerrit.config
+ type git
+ : OK
+ test -z
+ dirname bin/gerrit.sh
+ GERRIT_SITE_1=bin/..
+ test -f bin/../etc/gerrit.config
+ GERRIT_SITE=bin/..
+ test -z bin/..
+ pwd
+ INITIAL_DIR=/home/gerrit2/gerrit_install
+ cd bin/..
+ pwd
+ GERRIT_SITE=/home/gerrit2/gerrit_install
+ GERRIT_CONFIG=/home/gerrit2/gerrit_install/etc/gerrit.config
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test -r /home/gerrit2/gerrit_install/etc/gerrit.config
+ GERRIT_PID=/home/gerrit2/gerrit_install/logs/gerrit.pid
+ GERRIT_RUN=/home/gerrit2/gerrit_install/logs/gerrit.run
+ GERRIT_TMP=/home/gerrit2/gerrit_install/tmp
+ export GERRIT_TMP
+ JAVA_HOME_OLD=
+ get_config --get container.javaHome
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get container.javaHome
+ JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
+ test -z /usr/lib/jvm/java-7-openjdk-amd64/jre
+ test -z /usr/lib/jvm/java-7-openjdk-amd64/jre
+ test -z  -a -n /usr/lib/jvm/java-7-openjdk-amd64/jre -a -x /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -a ! -d /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+ JAVA=/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+ test -z /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+ test -z
+ JSTACK=/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/jstack
+ get_config --get-all container.javaOptions
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get-all = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get-all container.javaOptions
+ GERRIT_OPTIONS=
+ test -n
+ get_config --get container.heapLimit
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get container.heapLimit
+ GERRIT_MEMORY=
+ test -n
+ get_config --int core.packedGitOpenFiles
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--int = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --int core.packedGitOpenFiles
+ test x0 = x
+ echo
+ GERRIT_FDS=
+ test -z
+ GERRIT_FDS=128
+ expr 128 + 128
+ GERRIT_FDS=256
+ test 256 -lt 1024
+ GERRIT_FDS=1024
+ get_config --get container.user
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get container.user
+ GERRIT_USER=gerrit2
+ ulimit -c 0
+ ulimit -d unlimited
+ ulimit -f unlimited
+ ulimit -m
+ ulimit -m unlimited
+ ulimit -n 1024
+ ulimit -t unlimited
+ ulimit -v unlimited
+ ulimit -x
+ test -z
+ get_config --get container.war
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get container.war
+ GERRIT_WAR=
+ test -z
+ GERRIT_WAR=/home/gerrit2/gerrit_install/bin/gerrit.war
+ test -f /home/gerrit2/gerrit_install/bin/gerrit.war
+ test -z /home/gerrit2/gerrit_install/bin/gerrit.war -a -n gerrit2
+ test -z /home/gerrit2/gerrit_install/bin/gerrit.war
+ test -z gerrit2
+ RUN_ARGS=-jar /home/gerrit2/gerrit_install/bin/gerrit.war daemon -d /home/gerrit2/gerrit_install
+ get_config --bool container.slave
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--bool = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --bool container.slave
+ test  = true
+ get_config --get-all container.daemonOpt
+ test -f /home/gerrit2/gerrit_install/etc/gerrit.config
+ test x--get-all = x--int
+ git config --file /home/gerrit2/gerrit_install/etc/gerrit.config --get-all container.daemonOpt
+ DAEMON_OPTS=
+ test -n
+ test -n
+ test -x /usr/bin/perl
+ export JAVA
+ RUN_EXEC=/usr/bin/perl
+ RUN_Arg1=-e
+ RUN_Arg2=$x=$ENV{JAVA};exec $x @ARGV;die $!
+ RUN_Arg3=-- GerritCodeReview
+ printf %s Starting Gerrit Code Review:
Starting Gerrit Code Review: + test 1 = 0
+ test -z
+ sed -e s/^[^=]*=\([0-9]*\).*/\1/
+ UID=0
+ date +%s
+ RUN_ID=1461962528.4726
+ RUN_ARGS=-jar /home/gerrit2/gerrit_install/bin/gerrit.war daemon -d /home/gerrit2/gerrit_install --run-id=1461962528.4726
+ test 1 = 1
+ type start-stop-daemon
+ test 0 = 0
+ CH_USER=-c gerrit2
+ start-stop-daemon -S -b -c gerrit2 -p /home/gerrit2/gerrit_install/logs/gerrit.pid -m -d /home/gerrit2/gerrit_install -a /usr/bin/perl -- -e $x=$ENV{JAVA};exec $x @ARGV;die $! -- GerritCodeReview -jar /home/gerrit2/gerrit_install/bin/gerrit.war daemon -d /home/gerrit2/gerrit_install --run-id=1461962528.4726
+ : OK
+ test 0 = 0
+ cat /home/gerrit2/gerrit_install/logs/gerrit.pid
cat: /home/gerrit2/gerrit_install/logs/gerrit.pid: No such file or directory
+ PID=
+ test -f /proc//oom_score_adj
+ test -f /proc//oom_adj
+ TIMEOUT=90
+ sleep 1
+ running /home/gerrit2/gerrit_install/logs/gerrit.pid
+ test -f /home/gerrit2/gerrit_install/logs/gerrit.pid
+ cat /home/gerrit2/gerrit_install/logs/gerrit.pid
+ PID=4755
+ ps -p 4755
+ return 0
+ test 90 -gt 0
+ cat /home/gerrit2/gerrit_install/logs/gerrit.run
+ test x1461962528.4726 = x
+ sleep 2
+ expr 90 - 2
+ TIMEOUT=88
+ running /home/gerrit2/gerrit_install/logs/gerrit.pid
+ test -f /home/gerrit2/gerrit_install/logs/gerrit.pid
+ cat /home/gerrit2/gerrit_install/logs/gerrit.pid
+ PID=4755
+ ps -p 4755
+ return 0
+ test 88 -gt 0
+ cat /home/gerrit2/gerrit_install/logs/gerrit.run
+ test x1461962528.4726 = x
+ sleep 2
+ expr 88 - 2
The last 12 lines loop until the 90 second timeout reaches the end. After which the simple error FAILED appears. 
gerrit.config
# cat etc/gerrit.config
[gerrit]
    basePath = git
    canonicalWebUrl = http://gerrit.domain.com/
[database]
    type = mysql
    hostname = localhost
    database = reviewdb
    username = gerrit2
[index]
    type = LUCENE
[auth]
    type = HTTP
[receive]
    enableSignedPush = true
[sendemail]
    smtpServer = localhost
[container]
    user = gerrit2
    javaHome = /usr/lib/jvm/java-7-openjdk-amd64/jre
[sshd]
    listenAddress = *:29418
[httpd]
    listenUrl = proxy-http://127.0.0.1:8081/
[cache]
    directory = cache
ps ax | grep gerrit shows:
 4755 ?        Sl     0:10 GerritCodeReview -jar /home/gerrit2/gerrit_install/bin/gerrit.war daemon -d /home/gerrit2/gerrit_install --run-id=1461962528.4726
The Apache reverse proxy appears to be up and running correctly but is unable to connect on 127.0.0.1:8081:
[Fri Apr 29 16:28:56.820639 2016] [proxy_http:error] [pid 3658:tid 139954109798144] [client 74.56.105.94:49584] AH01114: HTTP: failed to make connection to backend: 127.0.0.1
[Fri Apr 29 16:28:57.573698 2016] [proxy:error] [pid 3658:tid 139954101405440] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8081 (127.0.0.1) failed
[Fri Apr 29 16:28:57.573766 2016] [proxy:error] [pid 3658:tid 139954101405440] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0.1) for 0s
Output from netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      992/sshd
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      1158/master
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1064/mysqld
tcp6       0      0 :::80                   :::*                    LISTEN      3655/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      992/sshd
tcp6       0      0 :::25                   :::*                    LISTEN      1158/master
tcp6       0      0 :::443                  :::*                    LISTEN      3655/apache2
I'm probably missing something very obvious, but don't see it. Anyone else have a suggestion what I might be missing?
Update: Starting from scratch with a minimum install, it would appear the issue is with the Bouncy Castle add ons. Gerrit 2.12.2 is asking for v1.52. Since Bouncy Castle only hosts the latest version 1.54, I used jar files from Maven.org:
http://repo2.maven.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.52/
http://repo2.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.52/
http://repo2.maven.org/maven2/org/bouncycastle/bcpg-jdk15on/1.52/
Once I install these after asking the enable signed push support, everything went back to the issue above. I'd still like to figure out how to get this working 
                How long did you wait? I've seen depoloyment where you needed to wait ~5minutes to get gerrir up&running..
– neutrinus
                Mar 28, 2017 at 12:37
                We had waited several minutes if I recall. We're running without bouncycastle for now and have no issues.
– Out of Control
                Mar 28, 2017 at 16:07
The problem might be lack of entropy. Gerrit reads from /dev/random, and on a headless server the entropy might be exhausted before Gerrit has had time to start up.
A fix could be to install the haveged package.
If you're on RHEL/CentOS/Fedora, the required commands would be:
yum install haveged
chkconfig haveged on
service haveged start
If you're on Debian/Ubuntu, the required commands would be:
apt-get install haveged
update-rc.d haveged defaults
service haveged start
After doing this, Gerrit should start relatively quickly.
Note: I've found this solution here.
        Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.