Red Hat Bugzilla – Bug 1181487
zookeeper service fails to start
Last modified: 2015-10-16 19:02:55 EDT
Description of problem:
zookeeper service fails to start.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. yum install zookeeper
2. cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
3. systemctl start zookeeper.service
Service exits with a failure.
Service keeps running
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-nop.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
[main] INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig - Reading configuration from: /etc/zookeeper/zoo.cfg
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - autopurge.snapRetainCount set to 3
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - autopurge.purgeInterval set to 0
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - Purge task is not scheduled.
[main] WARN org.apache.zookeeper.server.quorum.QuorumPeerMain - Either no config or no quorum defined in config, running in standalone mode
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.jmx.HierarchyDynamicMBean
at java.security.AccessController.doPrivileged(Native Method)
... 5 more
Did you properly configure zookeeper? It requires setup prior to run.
I believe I did. I tried to use the same configuration file with a downloaded tarball release and it worked just fine.
The above error looks like some dependencies are missing, because either java CLASSPATH is not set properly or they are not installed.
(In reply to Vaidas Jablonskis from comment #2)
> I believe I did. I tried to use the same configuration file with a
> downloaded tarball release and it worked just fine.
> The above error looks like some dependencies are missing, because either
> java CLASSPATH is not set properly or they are not installed.
I wouldn't be surprised if it's the classpath. zkCli.sh also fails because of classpath issues. bz#1189956
I think my change here fixes this: https://github.com/ctubbsii/zookeeper-fedora/commit/34ef8d1ec31fa96fdd51a36c3f95f1155200eadb
Would need verification. I may have missed something important from the classpath. I just tested starting 'zkServer.sh start' and connecting with 'zkCli.sh' with 'ls /'.
I don't have write access to push it, though. It also seems to address bz#1189956
A few other observations about this package I noticed:
1. It doesn't use standard launch scripts/env, such as what you get with %jpackage_script ; might be good to put zkEnv.sh in /etc/java/zookeeper.conf instead, with other standard env scripts; the upstream script names could be kept as aliases to these more standard ones (or vice-versa).
2. There seems to be an issue with jline... I'm guessing because the Fedora packaging hasn't been patched to work with jline2. Could backport the 3.5.0 fix from https://issues.apache.org/jira/browse/ZOOKEEPER-1718 into Fedora, since Fedora only ships JLine 2.
3. The default log4j configuration doesn't seem to work right (errors about no configured appenders... script might be missing a -Dlog4j=? log4j.properties on the classpath?). For the systemd service, I'd recommend using only logging to the console by default, and let systemd/journald take care of the logging.
I don't know how the package owner(s) want to deal with those... separate BZs?
It didn't work for me. Below is the output in /var/log/zookeeper/zookeeper.out.
I was able to make zookeeper happy by just appending few log4 jars to the horrendously log CLASSPATH in zkEbnv.sh. Is there primary GitHub repo that I can submit PR again?
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.quorum.QuorumPeerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.log4j.MDC.put(Ljava/lang/String;Ljava/lang/String;)V
I can confirm Christopher's patch in comment 4 resolves the issue for me for zookeeper-3.4.6-2.fc21.x86_64.
The patch in comment 4 looks insufficient to me. I haven't had a chance to revisit this, but I think there might be other dependencies that I missed in that patch. The method is sound, for discovering the classpath items, but I'd need to review the remaining runtime dependencies to ensure that the classpath is complete.
The issue mentioned in comment 5 was resolved via the following configuration before starting the server:
> export JVMFLAGS="-Dlog4j.configuration=file:///etc/zookeeper/log4j.properties"
The server start script does not seem to do this or anything to initializing log4j.
Otherwise, adding adding /etc/zookeeper/ to CLASSPATH will also resolve it.
Both quorum and multiple bindings jars errors are classpath issues addressed.
*** Bug 1189956 has been marked as a duplicate of this bug. ***
*** This bug has been marked as a duplicate of bug 1261458 ***