Bug 1261458 - Zookeeper service fails to start due to classpath errors, logging no error
Zookeeper service fails to start due to classpath errors, logging no error
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: zookeeper (Show other bugs)
22
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Christopher Tubbs
Fedora Extras Quality Assurance
: Reopened
: 1181487 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-09 07:35 EDT by KevinFarshaw
Modified: 2015-11-24 17:49 EST (History)
5 users (show)

See Also:
Fixed In Version: zookeeper-3.4.6-13.fc23 zookeeper-3.4.6-13.fc21 zookeeper-3.4.6-13.fc22 zookeeper-3.4.6-14.fc21
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-24 17:49:49 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patched /usr/libexec/zkEnv.sh (11.28 KB, application/x-shellscript)
2015-09-09 07:35 EDT, KevinFarshaw
no flags Details

  None (edit)
Description KevinFarshaw 2015-09-09 07:35:35 EDT
Created attachment 1071694 [details]
Patched /usr/libexec/zkEnv.sh

I've reviewed the other zookeeper issues, they report other problems and 
concern f21 rather the the current f22.

Description of problem:
After sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg,
Starting the zookeeper service fails silently without logging any error
to console or journal.

Manually starting it in the foreground using:

    sudo -u zookeeper /usr/bin/zkServer.sh start-foreground

Gives:

JMX enabled by default
Using config: /etc/zookeeper/zoo.cfg
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-simple.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/slf4j/slf4j-nop.jar!/org/slf4j/impl/StaticLoggerBinder.class]

Version-Release number of selected component (if applicable):
zookeeper-3.4.6-4.fc22.x86_64


How reproducible:
always

Steps to Reproduce:
1. sudo cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg
2. systemctl start zookeeper (fails, nothing logged)
3. sudo -u zookeeper  /usr/bin/zkServer.sh start-foreground (logs error to stderr)

Additional info:
Tipped off by https://issues.apache.org/jira/browse/SOLR-2369, it appears
that:
1. The classpath used to launch zk contains multiple jars of 
slog4j logging providers.
2. This is not allowed as documented by sl4fj at http://www.slf4j.org/codes.html#multiple_bindings
3. Inspection of the classpath setup by /usr/libexec/zkEnv.sh shows much
duplication, including multiple bindings.

Attached modified zkEnv.sh in which I've:
1. Removed duplicates.
2. Commented out alternate bindings and allegedly redundant jars. 
3. I did not intentionally remove any jars originally in the file (only commented).
4. Added individual log4j jars from appropriate package.

After updating zkEnv.sh, starting the service with systemctl complains
about missing precondition of file at /var/lib/zookeeper/data/myid.
After creating the file (with a single line containing an id such as
the digit '1'), zookeeper starts.

I have not exercised zookeeper to see whether something else is broken
(or was broken by the changes), but it does start up properly now.
Comment 1 KevinFarshaw 2015-09-11 12:10:37 EDT
Update: I Verified that basic zookeepr functionality works as expected (ls,mkdir, rm)

Also, it appears zk starts up fine without /var/lib/zookeeper/data/myid.
It would be more user-friendly IMO if "systemctl start zookeeper" simply worked out of the box, meaning only that cluster configuration was necessary only by
those that have one.

In my case, I only needed a running zk server to test another piece of software.
Comment 2 KevinFarshaw 2015-09-30 07:41:06 EDT
Bump.
Comment 3 KevinFarshaw 2015-10-07 01:59:19 EDT
Bump.
Comment 4 KevinFarshaw 2015-10-14 06:10:47 EDT
Bump. Please respond.
Comment 5 Timothy St. Clair 2015-10-14 10:15:48 EDT
sent email soliciting for co-maintainers for this package.
Comment 6 Christopher Tubbs 2015-10-14 11:46:18 EDT
I can take a look at this a bit later today.
Comment 7 Christopher Tubbs 2015-10-14 17:02:13 EDT
Sorry, other stuff came up today. I will make this the first thing I look at tomorrow. These bugs have been annoying me, too.
Comment 8 Christopher Tubbs 2015-10-16 17:24:47 EDT
I reviewed and applied the patch and have kicked off some builds for f21 through rawhide. If all goes well, I'll push some updates to test soon.
Comment 9 KevinFarshaw 2015-10-16 17:55:21 EDT
Thanks. I'll test and leave karma when you push. Did you get rid of the myid
precondition in the service file too?
Comment 10 Christopher Tubbs 2015-10-16 18:58:08 EDT
(In reply to KevinFarshaw from comment #9)
> Thanks. I'll test and leave karma when you push. Did you get rid of the myid
> precondition in the service file too?

I did not. I'd rather investigate that separately from the classpath issue, because I'm not too sure about why that precondition was added. Like, is that to ensure ZK doesn't accidentally join the wrong quorum?
Comment 11 Christopher Tubbs 2015-10-16 19:02:55 EDT
*** Bug 1181487 has been marked as a duplicate of this bug. ***
Comment 12 Fedora Update System 2015-10-16 19:07:04 EDT
zookeeper-3.4.6-11.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-d71fcc2e48
Comment 13 Fedora Update System 2015-10-16 19:08:11 EDT
zookeeper-3.4.6-11.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-bd2a25f292
Comment 14 Fedora Update System 2015-10-16 19:08:58 EDT
zookeeper-3.4.6-11.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-970a41de03
Comment 15 KevinFarshaw 2015-10-17 08:35:54 EDT
(In reply to Christopher Tubbs from comment #10)
> (In reply to KevinFarshaw from comment #9)
> > Thanks. I'll test and leave karma when you push. Did you get rid of the myid
> > precondition in the service file too?
> 
> I did not. I'd rather investigate that separately from the classpath issue,
> because I'm not too sure about why that precondition was added. Like, is
> that to ensure ZK doesn't accidentally join the wrong quorum?

That means the server will still be unrunnable in the default install.

I've looked at the SRPM repo. 

http://pkgs.fedoraproject.org/git/zookeeper.git

There's nothing in the spec file or git log that indicates this was a specific fix for a problem/race/danger. The commits adding the service files was

94aec27 Jul 18 2013 Add a server subpackage.  Jeffrey C. Ollie  

commited by tstclair. The relevent hunk in the diff for the service file was:

--- a/zookeeper.service
+++ b/zookeeper.service
@@ -1,14 +1,20 @@
 [Unit]
-Description=Zookeeper
+Description=Apache ZooKeeper
+After=network.target
+ConditionPathExists=/etc/zookeeper/zoo.cfg
+ConditionPathExists=/etc/zookeeper/log4j.properties
+ConditionPathExists=/var/lib/zookeeper/data/myid

I am not an authority on zookeeper but, as I read  the manual
https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper

the myid file is only required when zookeeper is explicitly
configured for quorum operation, in which case it should match 
one of the quorum member entries added to /etc/zookeeper/zoo.cfg.

on fedora (22):
$ rpm -qlv zookeeper
   0 Feb 16  2015 /etc/zookeeper
2161 Mar 12  2014 /etc/zookeeper/log4j.properties
   0 Feb 16  2015 /etc/zookeeper/zoo.cfg
 965 Feb 16  2015 /etc/zookeeper/zoo_sample.cfg

neither the empty default config, nor the sample config provided alongside
it define a quorum, they are both geared for standalone operation. Since

1. myid is not needed in standalone operation.
2. the default install will always come up in standalone operation
unless someone manually edits the configuration and adds quorum entries.

I think it is reasonable to remove that precondition and require that 
a person switching the daemon from standalone mode to quorum mode
do that properly, following the procedure described in the zk documentation.

It would be my pleasure to open a new issue for this and move this over there,
I'm just not eager to spend an addtional two months bumping that one every week.
Comment 16 Christopher Tubbs 2015-10-17 08:44:57 EDT
I agree with your reasoning and will modify or remove that precondition. Let's proceed in a separate issue, only so we can track the history better later if we need to.
Comment 17 KevinFarshaw 2015-10-17 14:19:34 EDT
(In reply to Christopher Tubbs from comment #16)
> I agree with your reasoning and will modify or remove that precondition.
> Let's proceed in a separate issue, only so we can track the history better
> later if we need to.

Thank you. Opened #1272694 and copied the context over.
Comment 18 Fedora Update System 2015-10-17 14:21:05 EDT
zookeeper-3.4.6-11.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-970a41de03
Comment 19 Fedora Update System 2015-10-17 20:18:27 EDT
zookeeper-3.4.6-11.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-d71fcc2e48
Comment 20 Fedora Update System 2015-10-17 20:52:43 EDT
zookeeper-3.4.6-11.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-bd2a25f292
Comment 21 Fedora Update System 2015-10-19 21:32:06 EDT
zookeeper-3.4.6-12.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-3467dc710b
Comment 22 Fedora Update System 2015-10-19 21:32:32 EDT
zookeeper-3.4.6-12.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-93bded3672
Comment 23 Fedora Update System 2015-10-19 21:33:02 EDT
zookeeper-3.4.6-12.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15
Comment 24 Fedora Update System 2015-10-20 17:58:05 EDT
zookeeper-3.4.6-12.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-3467dc710b
Comment 25 Fedora Update System 2015-10-26 06:29:26 EDT
zookeeper-3.4.6-12.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15
Comment 26 Fedora Update System 2015-10-26 14:32:07 EDT
zookeeper-3.4.6-12.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-93bded3672
Comment 27 Fedora Update System 2015-11-03 17:29:59 EST
zookeeper-3.4.6-13.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-45fe2ff2ac
Comment 28 Fedora Update System 2015-11-03 17:30:29 EST
zookeeper-3.4.6-13.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-eef6ca6bf6
Comment 29 Fedora Update System 2015-11-03 17:31:04 EST
zookeeper-3.4.6-13.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-ef5e68cbf8
Comment 30 Fedora Update System 2015-11-04 19:22:52 EST
zookeeper-3.4.6-13.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-ef5e68cbf8
Comment 31 Fedora Update System 2015-11-04 20:28:40 EST
zookeeper-3.4.6-13.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-45fe2ff2ac
Comment 32 Fedora Update System 2015-11-05 10:53:06 EST
zookeeper-3.4.6-13.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-eef6ca6bf6
Comment 33 Fedora Update System 2015-11-13 17:55:22 EST
zookeeper-3.4.6-13.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 34 Fedora Update System 2015-11-13 20:51:33 EST
zookeeper-3.4.6-13.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
Comment 35 Fedora Update System 2015-11-13 20:53:07 EST
zookeeper-3.4.6-13.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
Comment 36 Fedora Update System 2015-11-15 03:36:58 EST
zookeeper-3.4.6-12.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-2dc6711b15
Comment 37 Christopher Tubbs 2015-11-15 03:39:35 EST
Ignore the messages for zookeeper-3.4.6-12.fc21 update. It's in a strange state in Bodhi.
Comment 38 Fedora Update System 2015-11-15 03:47:08 EST
zookeeper-3.4.6-14.fc21 has been submitted as an update to Fedora 21. https://bodhi.fedoraproject.org/updates/FEDORA-2015-b1d473beb0
Comment 39 Fedora Update System 2015-11-16 01:52:19 EST
zookeeper-3.4.6-14.fc21 has been pushed to the Fedora 21 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update zookeeper'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-b1d473beb0
Comment 40 Fedora Update System 2015-11-24 17:49:43 EST
zookeeper-3.4.6-14.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.