Bug 1181487 - zookeeper service fails to start
zookeeper service fails to start
Status: CLOSED DUPLICATE of bug 1261458
Product: Fedora
Classification: Fedora
Component: zookeeper (Show other bugs)
21
All Linux
low Severity low
: ---
: ---
Assigned To: Timothy St. Clair
Fedora Extras Quality Assurance
:
: 1189956 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-01-13 04:55 EST by Vaidas Jablonskis
Modified: 2015-10-16 19:02 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-16 19:02:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vaidas Jablonskis 2015-01-13 04:55:13 EST
Description of problem:
zookeeper service fails to start.

Version-Release number of selected component (if applicable):
3.4.6-2.fc21

How reproducible:
Always

Steps to Reproduce:
1. yum install zookeeper
2. cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
3. systemctl start zookeeper.service

Actual results:
Service exits with a failure.

Expected results:
Service keeps running

Additional info:
cat /var/log/zookeeper/zookeeper.out 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-nop.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
[main] INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig - Reading configuration from: /etc/zookeeper/zoo.cfg
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - autopurge.snapRetainCount set to 3
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - autopurge.purgeInterval set to 0
[main] INFO org.apache.zookeeper.server.DatadirCleanupManager - Purge task is not scheduled.
[main] WARN org.apache.zookeeper.server.quorum.QuorumPeerMain - Either no config or no quorum defined in config, running  in standalone mode
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
	at org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:50)
	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:74)
	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.jmx.HierarchyDynamicMBean
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 5 more
Comment 1 Timothy St. Clair 2015-01-13 08:34:42 EST
Did you properly configure zookeeper?  It requires setup prior to run.
Comment 2 Vaidas Jablonskis 2015-01-13 08:50:11 EST
I believe I did. I tried to use the same configuration file with a downloaded tarball release and it worked just fine.

The above error looks like some dependencies are missing, because either java CLASSPATH is not set properly or they are not installed.
Comment 3 Christopher Tubbs 2015-02-05 19:25:11 EST
(In reply to Vaidas Jablonskis from comment #2)
> I believe I did. I tried to use the same configuration file with a
> downloaded tarball release and it worked just fine.
> 
> The above error looks like some dependencies are missing, because either
> java CLASSPATH is not set properly or they are not installed.

I wouldn't be surprised if it's the classpath. zkCli.sh also fails because of classpath issues. bz#1189956
Comment 4 Christopher Tubbs 2015-02-07 00:40:38 EST
I think my change here fixes this: https://github.com/ctubbsii/zookeeper-fedora/commit/34ef8d1ec31fa96fdd51a36c3f95f1155200eadb

Would need verification. I may have missed something important from the classpath. I just tested starting 'zkServer.sh start' and connecting with 'zkCli.sh' with 'ls /'.

I don't have write access to push it, though. It also seems to address bz#1189956

A few other observations about this package I noticed:

1. It doesn't use standard launch scripts/env, such as what you get with %jpackage_script ; might be good to put zkEnv.sh in /etc/java/zookeeper.conf instead, with other standard env scripts; the upstream script names could be kept as aliases to these more standard ones (or vice-versa).

2. There seems to be an issue with jline... I'm guessing because the Fedora packaging hasn't been patched to work with jline2. Could backport the 3.5.0 fix from https://issues.apache.org/jira/browse/ZOOKEEPER-1718 into Fedora, since Fedora only ships JLine 2.

3. The default log4j configuration doesn't seem to work right (errors about no configured appenders... script might be missing a -Dlog4j=? log4j.properties on the classpath?). For the systemd service, I'd recommend using only logging to the console by default, and let systemd/journald take care of the logging.

I don't know how the package owner(s) want to deal with those... separate BZs?
Comment 5 Misho Krastev 2015-02-10 08:06:57 EST
It didn't work for me. Below is the output in /var/log/zookeeper/zookeeper.out.

I was able to make zookeeper happy by just appending few log4 jars to the horrendously log CLASSPATH in zkEbnv.sh. Is there primary GitHub repo that I can submit PR again?

log4j:WARN No appenders could be found for logger (org.apache.zookeeper.server.quorum.QuorumPeerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.log4j.MDC.put(Ljava/lang/String;Ljava/lang/String;)V
	at org.slf4j.impl.Log4jMDCAdapter.put(Log4jMDCAdapter.java:59)
	at org.slf4j.MDC.put(MDC.java:116)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:362)
	at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Comment 6 [DISABLED] 2015-03-17 12:35:05 EDT
I can confirm Christopher's patch in comment 4 resolves the issue for me for zookeeper-3.4.6-2.fc21.x86_64.
Comment 7 Christopher Tubbs 2015-03-17 13:12:48 EDT
The patch in comment 4 looks insufficient to me. I haven't had a chance to revisit this, but I think there might be other dependencies that I missed in that patch. The method is sound, for discovering the classpath items, but I'd need to review the remaining runtime dependencies to ensure that the classpath is complete.
Comment 8 [DISABLED] 2015-03-17 20:28:56 EDT
The issue mentioned in comment 5 was resolved via the following configuration before starting the server:

> export JVMFLAGS="-Dlog4j.configuration=file:///etc/zookeeper/log4j.properties"

The server start script does not seem to do this or anything to initializing log4j.
Comment 9 [DISABLED] 2015-03-17 21:37:38 EDT
Otherwise, adding adding /etc/zookeeper/ to CLASSPATH will also resolve it.
Comment 10 KevinFarshaw 2015-09-09 07:37:06 EDT
related 1261458. 

Both quorum and multiple bindings jars errors are classpath issues addressed.
Comment 11 Christopher Tubbs 2015-10-16 19:01:32 EDT
*** Bug 1189956 has been marked as a duplicate of this bug. ***
Comment 12 Christopher Tubbs 2015-10-16 19:02:55 EDT

*** This bug has been marked as a duplicate of bug 1261458 ***

Note You need to log in before you can comment on or make changes to this bug.