Bug 1261458

Summary: Zookeeper service fails to start due to classpath errors, logging no error
Product: [Fedora] Fedora Reporter: KevinFarshaw
Component: zookeeperAssignee: Christopher Tubbs <ctubbsii>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: ctubbsii, ethan, java-sig-commits, s, tstclair
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: zookeeper-3.4.6-13.fc23 zookeeper-3.4.6-13.fc21 zookeeper-3.4.6-13.fc22 zookeeper-3.4.6-14.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-24 22:49:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patched /usr/libexec/zkEnv.sh none

Description KevinFarshaw 2015-09-09 11:35:35 UTC
Created attachment 1071694 [details]
Patched /usr/libexec/zkEnv.sh

I've reviewed the other zookeeper issues, they report other problems and 
concern f21 rather the the current f22.

Description of problem:
After sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg,
Starting the zookeeper service fails silently without logging any error
to console or journal.

Manually starting it in the foreground using:

    sudo -u zookeeper /usr/bin/zkServer.sh start-foreground

Gives:

JMX enabled by default
Using config: /etc/zookeeper/zoo.cfg
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-nop.jar!/org/slf4j/impl/StaticLoggerBinder.class]

Version-Release number of selected component (if applicable):
zookeeper-3.4.6-4.fc22.x86_64


How reproducible:
always

Steps to Reproduce:
1. sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
2. systemctl start zookeeper (fails, nothing logged)
3. sudo -u zookeeper  /usr/bin/zkServer.sh start-foreground (logs error to stderr)

Additional info:
Tipped off by https://issues.apache.org/jira/browse/SOLR-2369, it appears
that:
1. The classpath used to launch zk contains multiple jars of 
slog4j logging providers.
2. This is not allowed as documented by sl4fj at http://www.slf4j.org/codes.html#multiple_bindings
3. Inspection of the classpath setup by /usr/libexec/zkEnv.sh shows much
duplication, including multiple bindings.

Attached modified zkEnv.sh in which I've:
1. Removed duplicates.
2. Commented out alternate bindings and allegedly redundant jars. 
3. I did not intentionally remove any jars originally in the file (only commented).
4. Added individual log4j jars from appropriate package.

After updating zkEnv.sh, starting the service with systemctl complains
about missing precondition of file at /var/lib/zookeeper/data/myid.
After creating the file (with a single line containing an id such as
the digit '1'), zookeeper starts.

I have not exercised zookeeper to see whether something else is broken
(or was broken by the changes), but it does start up properly now.

Comment 1 KevinFarshaw 2015-09-11 16:10:37 UTC
Update: I Verified that basic zookeepr functionality works as expected (ls,mkdir, rm)

Also, it appears zk starts up fine without /var/lib/zookeeper/data/myid.
It would be more user-friendly IMO if "systemctl start zookeeper" simply worked out of the box, meaning only that cluster configuration was necessary only by
those that have one.

In my case, I only needed a running zk server to test another piece of software.

Comment 2 KevinFarshaw 2015-09-30 11:41:06 UTC
Bump.

Comment 3 KevinFarshaw 2015-10-07 05:59:19 UTC
Bump.

Comment 4 KevinFarshaw 2015-10-14 10:10:47 UTC
Bump. Please respond.

Comment 5 Timothy St. Clair 2015-10-14 14:15:48 UTC
sent email soliciting for co-maintainers for this package.

Comment 6 Christopher Tubbs 2015-10-14 15:46:18 UTC
I can take a look at this a bit later today.

Comment 7 Christopher Tubbs 2015-10-14 21:02:13 UTC
Sorry, other stuff came up today. I will make this the first thing I look at tomorrow. These bugs have been annoying me, too.

Comment 8 Christopher Tubbs 2015-10-16 21:24:47 UTC
I reviewed and applied the patch and have kicked off some builds for f21 through rawhide. If all goes well, I'll push some updates to test soon.

Comment 9 KevinFarshaw 2015-10-16 21:55:21 UTC
Thanks. I'll test and leave karma when you push. Did you get rid of the myid
precondition in the service file too?

Comment 10 Christopher Tubbs 2015-10-16 22:58:08 UTC
(In reply to KevinFarshaw from comment #9)
> Thanks. I'll test and leave karma when you push. Did you get rid of the myid
> precondition in the service file too?

I did not. I'd rather investigate that separately from the classpath issue, because I'm not too sure about why that precondition was added. Like, is that to ensure ZK doesn't accidentally join the wrong quorum?

Comment 11 Christopher Tubbs 2015-10-16 23:02:55 UTC
*** Bug 1181487 has been marked as a duplicate of this bug. ***

Comment 12 Fedora Update System 2015-10-16 23:07:04 UTC
zookeeper-3.4.6-11.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d71fcc2e48

Comment 13 Fedora Update System 2015-10-16 23:08:11 UTC
zookeeper-3.4.6-11.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-bd2a25f292

Comment 14 Fedora Update System 2015-10-16 23:08:58 UTC
zookeeper-3.4.6-11.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-970a41de03

Comment 15 KevinFarshaw 2015-10-17 12:35:54 UTC
(In reply to Christopher Tubbs from comment #10)
> (In reply to KevinFarshaw from comment #9)
> > Thanks. I'll test and leave karma when you push. Did you get rid of the myid
> > precondition in the service file too?
> 
> I did not. I'd rather investigate that separately from the classpath issue,
> because I'm not too sure about why that precondition was added. Like, is
> that to ensure ZK doesn't accidentally join the wrong quorum?

That means the server will still be unrunnable in the default install.

I've looked at the SRPM repo. 

http://pkgs.fedoraproject.org/git/zookeeper.git

There's nothing in the spec file or git log that indicates this was a specific fix for a problem/race/danger. The commits adding the service files was

94aec27 Jul 18 2013 Add a server subpackage.  Jeffrey C. Ollie  

commited by tstclair. The relevent hunk in the diff for the service file was:

--- a/zookeeper.service
+++ b/zookeeper.service
@@ -1,14 +1,20 @@
 [Unit]
-Description=Zookeeper
+Description=Apache ZooKeeper
+After=network.target
+ConditionPathExists=/etc/zookeeper/zoo.cfg
+ConditionPathExists=/etc/zookeeper/log4j.properties
+ConditionPathExists=/var/lib/zookeeper/data/myid

I am not an authority on zookeeper but, as I read  the manual
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

the myid file is only required when zookeeper is explicitly
configured for quorum operation, in which case it should match 
one of the quorum member entries added to /etc/zookeeper/zoo.cfg.

on fedora (22):
$ rpm -qlv zookeeper
   0 Feb 16  2015 /etc/zookeeper
2161 Mar 12  2014 /etc/zookeeper/log4j.properties
   0 Feb 16  2015 /etc/zookeeper/zoo.cfg
 965 Feb 16  2015 /etc/zookeeper/zoo_sample.cfg

neither the empty default config, nor the sample config provided alongside
it define a quorum, they are both geared for standalone operation. Since

1. myid is not needed in standalone operation.
2. the default install will always come up in standalone operation
unless someone manually edits the configuration and adds quorum entries.

I think it is reasonable to remove that precondition and require that 
a person switching the daemon from standalone mode to quorum mode
do that properly, following the procedure described in the zk documentation.

It would be my pleasure to open a new issue for this and move this over there,
I'm just not eager to spend an addtional two months bumping that one every week.

Comment 16 Christopher Tubbs 2015-10-17 12:44:57 UTC
I agree with your reasoning and will modify or remove that precondition. Let's proceed in a separate issue, only so we can track the history better later if we need to.

Comment 17 KevinFarshaw 2015-10-17 18:19:34 UTC
(In reply to Christopher Tubbs from comment #16)
> I agree with your reasoning and will modify or remove that precondition.
> Let's proceed in a separate issue, only so we can track the history better
> later if we need to.

Thank you. Opened #1272694 and copied the context over.

Comment 18 Fedora Update System 2015-10-17 18:21:05 UTC
zookeeper-3.4.6-11.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-970a41de03

Comment 19 Fedora Update System 2015-10-18 00:18:27 UTC
zookeeper-3.4.6-11.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d71fcc2e48

Comment 20 Fedora Update System 2015-10-18 00:52:43 UTC
zookeeper-3.4.6-11.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-bd2a25f292

Comment 21 Fedora Update System 2015-10-20 01:32:06 UTC
zookeeper-3.4.6-12.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-3467dc710b

Comment 22 Fedora Update System 2015-10-20 01:32:32 UTC
zookeeper-3.4.6-12.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-93bded3672

Comment 23 Fedora Update System 2015-10-20 01:33:02 UTC
zookeeper-3.4.6-12.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15

Comment 24 Fedora Update System 2015-10-20 21:58:05 UTC
zookeeper-3.4.6-12.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-3467dc710b

Comment 25 Fedora Update System 2015-10-26 10:29:26 UTC
zookeeper-3.4.6-12.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15

Comment 26 Fedora Update System 2015-10-26 18:32:07 UTC
zookeeper-3.4.6-12.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-93bded3672

Comment 27 Fedora Update System 2015-11-03 22:29:59 UTC
zookeeper-3.4.6-13.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-45fe2ff2ac

Comment 28 Fedora Update System 2015-11-03 22:30:29 UTC
zookeeper-3.4.6-13.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-eef6ca6bf6

Comment 29 Fedora Update System 2015-11-03 22:31:04 UTC
zookeeper-3.4.6-13.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-ef5e68cbf8

Comment 30 Fedora Update System 2015-11-05 00:22:52 UTC
zookeeper-3.4.6-13.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-ef5e68cbf8

Comment 31 Fedora Update System 2015-11-05 01:28:40 UTC
zookeeper-3.4.6-13.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-45fe2ff2ac

Comment 32 Fedora Update System 2015-11-05 15:53:06 UTC
zookeeper-3.4.6-13.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-eef6ca6bf6

Comment 33 Fedora Update System 2015-11-13 22:55:22 UTC
zookeeper-3.4.6-13.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 34 Fedora Update System 2015-11-14 01:51:33 UTC
zookeeper-3.4.6-13.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.

Comment 35 Fedora Update System 2015-11-14 01:53:07 UTC
zookeeper-3.4.6-13.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.

Comment 36 Fedora Update System 2015-11-15 08:36:58 UTC
zookeeper-3.4.6-12.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15

Comment 37 Christopher Tubbs 2015-11-15 08:39:35 UTC
Ignore the messages for zookeeper-3.4.6-12.fc21 update. It's in a strange state in Bodhi.

Comment 38 Fedora Update System 2015-11-15 08:47:08 UTC
zookeeper-3.4.6-14.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-b1d473beb0

Comment 39 Fedora Update System 2015-11-16 06:52:19 UTC
zookeeper-3.4.6-14.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-b1d473beb0

Comment 40 Fedora Update System 2015-11-24 22:49:43 UTC
zookeeper-3.4.6-14.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.