Bug 1148604

Summary:	rabbitmq-server does not start due to missing symbols
Product:	[Fedora] Fedora	Reporter:	Lars Kellogg-Stedman <lars>
Component:	erlang-sd_notify	Assignee:	John Eckersberg <jeckersb>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	21	CC:	chdent, dustymabe, erlang, hubert.plociniczak, jeckersb, lemenkov, matthias, rjones, s
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	erlang-sd_notify-0.1-4.fc21	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-01 16:55:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lars Kellogg-Stedman 2014-10-01 20:23:20 UTC

Trying to start rabbitmq-server (rabbitmq-server-3.1.5-9.fc21.noarch) yields the following:

# /sbin/service rabbitmq-server start
Redirecting to /bin/systemctl start  rabbitmq-server.service
Job for rabbitmq-server.service failed. See 'systemctl status rabbitmq-server.service' and 'journalctl -xn' for details.

Running "journalctl -u rabbitmq-server" yields:

Oct 01 20:16:53 f21-os-test systemd[1]: Starting RabbitMQ broker...
Oct 01 20:16:53 f21-os-test systemd[1]: rabbitmq-server.service: Got notification message from PID 14545, but reception only permitted for main PID 14532
Oct 01 20:16:53 f21-os-test systemd[1]: rabbitmq-server.service: Got notification message from PID 14547, but reception only permitted for main PID 14532
Oct 01 20:16:53 f21-os-test systemd[1]: rabbitmq-server.service: Got notification message from PID 14571, but reception only permitted for main PID 14532
Oct 01 20:16:53 f21-os-test systemd[1]: rabbitmq-server.service: Got notification message from PID 14572, but reception only permitted for main PID 14532
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: RabbitMQ 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc.
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: ##  ##
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: ##########  Logs: /var/log/rabbitmq/rabbit
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: ######  ##        /var/log/rabbitmq/rabbit
Oct 01 20:16:54 f21-os-test rabbitmq-server[14532]: ##########
Oct 01 20:16:55 f21-os-test rabbitmq-server[14532]: Starting broker... completed with 0 plugins.
Oct 01 20:18:23 f21-os-test systemd[1]: rabbitmq-server.service start operation timed out. Terminating.
Oct 01 20:18:23 f21-os-test systemd[1]: Failed to start RabbitMQ broker.
Oct 01 20:18:23 f21-os-test systemd[1]: Unit rabbitmq-server.service entered failed state.


/var/log/rabbitmq/hostname.log contains:

=INFO REPORT==== 1-Oct-2014::20:16:54 ===
Starting RabbitMQ 3.1.5 on Erlang 17
Copyright (C) 2007-2013 GoPivotal, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 1-Oct-2014::20:16:54 ===
node           : rabbit@f21-os-test
home dir       : /var/lib/rabbitmq
cookie hash    : PZPVSneawENzOO4Ft5aeuA==
log            : /var/log/rabbitmq/rabbit
sasl log       : /var/log/rabbitmq/rabbit
database dir   : /var/lib/rabbitmq/mnesia/rabbit@f21-os-test

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
Limiting to approx 924 file handles (829 sockets)

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
Memory limit set to 1581MB of 3953MB total.

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
Disk free limit set to 1000MB

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=WARNING REPORT==== 1-Oct-2014::20:16:55 ===
msg_store_persistent: rebuilding indices from scratch

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
started TCP Listener on [::]:5672

=WARNING REPORT==== 1-Oct-2014::20:16:55 ===
The on_load function for module sd_notify returned {error,
                                                    {load_failed,
                                                     "Failed to load NIF library: '/usr/lib64/erlang/lib/sd_notify-0.1/priv/sd_notify_drv.so: undefined symbol: sd_notify'"}}

=INFO REPORT==== 1-Oct-2014::20:16:55 ===
Server startup complete; 0 plugins started.

Comment 1 John Eckersberg 2014-10-01 20:40:24 UTC

I think possibly erlang-sd_notify needs rebuilt in rawhide following the most recent erlang rebase?

Comment 2 John Eckersberg 2014-10-02 14:32:51 UTC

OK, what really is happening is that the sd_notify_drv.so is not linked against libsystem-daemon.  This used to work previously because something else in the erlang runtime was linked against libsystemd-daemon, so the symbol was already mapped into the process and loading the sd_notify module would work.

However the newer erlang builds don't link with libsystemd-daemon necessarily, which is what happens here.

I'm updating erlang-sd_notify to explicitly carry that link-time dependency.

Comment 3 Peter Lemenkov 2014-10-02 14:41:42 UTC

(In reply to John Eckersberg from comment #2)
> OK, what really is happening is that the sd_notify_drv.so is not linked
> against libsystem-daemon.  This used to work previously because something
> else in the erlang runtime was linked against libsystemd-daemon, so the
> symbol was already mapped into the process and loading the sd_notify module
> would work.
> 
> However the newer erlang builds don't link with libsystemd-daemon
> necessarily, which is what happens here.
> 
> I'm updating erlang-sd_notify to explicitly carry that link-time dependency.

John, thanks for digging into this. I added you as a co-maintainers. Looks like this change was related to a recent libsystemd-daemon.so -> libsystemd.so change (see this #1125105).

Comment 4 Peter Lemenkov 2014-10-02 14:51:18 UTC

(In reply to John Eckersberg from comment #2)
> OK, what really is happening is that the sd_notify_drv.so is not linked
> against libsystem-daemon.  This used to work previously because something
> else in the erlang runtime was linked against libsystemd-daemon, so the
> symbol was already mapped into the process and loading the sd_notify module
> would work.
> 
> However the newer erlang builds don't link with libsystemd-daemon
> necessarily, which is what happens here.
> 
> I'm updating erlang-sd_notify to explicitly carry that link-time dependency.

Sadly but Ive made a mistake. I didn't even used -lsystemd while building:

https://github.com/lemenkov/erlang-sd_notify

In fact this was just a coincidence that my code is worked somehow (erlang already mapped some systemd functions as you found out). In order to fix this I believe that erlang-sd_notify requires rebar.config file with something like this:

https://github.com/basho/ebloom/blob/develop/rebar.config

This would tell rebar what libraries should this application be linked against. Also it's possible to setup some other variables here (CFLAGS, CXXFLAGS, etc).

Comment 5 John Eckersberg 2014-10-02 14:59:30 UTC

The patch I pushed into git a moment ago just sets LDFLAGS=-lsystemd when invoking rebar in the spec.

Comment 6 Peter Lemenkov 2014-10-02 15:03:49 UTC

(In reply to John Eckersberg from comment #5)
> The patch I pushed into git a moment ago just sets LDFLAGS=-lsystemd when
> invoking rebar in the spec.

Looks good to me. Don't forget to merge in into f21.

Comment 7 Fedora Update System 2014-10-02 15:26:33 UTC

erlang-sd_notify-0.1-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/erlang-sd_notify-0.1-4.fc21

Comment 8 Dusty Mabe 2014-10-02 16:47:23 UTC

As a workaround you can use the following:

# If erlang-sd_notify-0.1-4 is in the repos then use that.. 
# If not install directly from koji
# NOTE: the link is for x86_64 rpm

yum install -y erlang-sd_notify-0.1-4 || yum install http://kojipkgs.fedoraproject.org/packages/erlang-sd_notify/0.1/4.fc21/data/signed/95a43f54/x86_64/erlang-sd_notify-0.1-4.fc21.x86_64.rpm

Comment 9 Dusty Mabe 2014-10-02 17:47:04 UTC

Just a thought here. Should rabbitmq-server be updated to require erlang-sd_notify >= 0.1-4 ?

Comment 10 John Eckersberg 2014-10-02 18:06:29 UTC

(In reply to Dusty Mabe from comment #9)
> Just a thought here. Should rabbitmq-server be updated to require
> erlang-sd_notify >= 0.1-4 ?

I don't think so.  If a user is going to update rabbitmq-server, they'll most likely get it by running "yum update", which would also pull in the updated erlang-sd_notify.  Also, rabbitmq-server can function correctly even with erlang-sd_notify=0.1.3, as long as an older version of erlang is used.

Comment 11 Fedora Update System 2014-10-03 03:56:49 UTC

Package erlang-sd_notify-0.1-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing erlang-sd_notify-0.1-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-11943/erlang-sd_notify-0.1-4.fc21
then log in and leave karma (feedback).

Comment 12 Fedora Update System 2014-11-01 16:55:10 UTC

erlang-sd_notify-0.1-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 13 Matthias Saou 2015-03-17 10:16:58 UTC

Quick note in case anyone bumps into this and finds this report : On RHEL7 the "-lsystemd" doesn't work, that library isn't found. With "-lsystemd-daemon" the packages rebuild and work fine.