Bug 589683

Summary: Broker misconfiguration causes broker stop to fail
Product: Red Hat Enterprise MRG Reporter: Kim van der Riet <kim.vdriet>
Component: qpid-cppAssignee: Kim van der Riet <kim.vdriet>
Status: CLOSED UPSTREAM QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: DevelopmentCC: gsim, jross
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-15 19:15:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kim van der Riet 2010-05-06 17:10:31 UTC
If the broker configuration file /etc/qpidd.conf is misconfigured, the broker will start, but it cannot be stopped again (using service stop).

Starting with clean /var/lib/qpidd and /var/run/qpidd dirs:

[root@mrg10 ~]# cat /etc/qpidd.conf 
cluster-mechanism=ANONYMOUS
auth=no
log-enable=info+
log-to-file=test.log
[root@mrg10 ~]# service qpidd start
Starting Qpid AMQP daemon: 2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
                                                           [  OK  ]
Unexpected error: Can't open log file: test.log
[root@mrg10 ~]# service qpidd stop
Stopping Qpid AMQP daemon:                                 [FAILED]
[root@mrg10 ~]# ps -eaf | grep qpidd
qpidd     4870     1  0 12:50 ?        00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
root      4899  3322  0 12:52 pts/0    00:00:00 grep qpidd

NOTE 1: In spite of the broker error above, the log file is in fact created in the defualt directory (/var/lib/qpidd/test.log).

NOTE 2: Changing the config file to an absolute path (log-to-file=/var/lib/qpidd/test.log) fixes the broker startup error; in this case the service stop works ok.

NOTE 3: Both these cases place a pid file /var/run/qpidd/qpidd.5672.pid which contains a binary PID number. However, if the config file is fixed as in NOTE 2, then in addition a second pid file is created: /var/run/qpidd.pid which contains a string representation of the PID. It is likely this second file which causes the service stop to fail.

Comment 1 Kim van der Riet 2010-05-06 17:12:23 UTC
Tested on RHEL-6 using qpid-cpp-server-0.7.935473-1.el6.x86_64 and its dependents.

Comment 2 Kim van der Riet 2010-06-10 19:49:55 UTC
This is a much narrower, but more complex case than I initially thought.

The use of the "log-to-file" option in the config file is the source of this error. Genuine configuration file errors are either ignored or result in a failure to start the daemon in the first place.

When the broker daemon is started, it reads the config file and opens the log file as expected. This also creates an empty pid file. The startup script then uses "qpidd --check" to get the PID and write it into the empty pid file. This instance of qpidd reads the same config file and then attempts to open the same log file as the broker itself, causing it to fail with the message "Unexpected error: Can't open log file: test.log" (in the example above). This failure in turn leaves the pid file empty, and hence the use of "service stop" fails.

There is no simple solution to this that is not "hacky".

Ideally, this should be solved within qpidd itself - for instance if the daemon service options --quit or --check are used, then logging and module loading should be disabled.

Comment 3 Kim van der Riet 2010-06-11 12:53:30 UTC
Ignore analysis in comment #2 above - incorrect conclusion.

Absolute paths work ok. It turns out that using log-to-file without an absolute path causes the "qpidd --check" to run in the incorrect directory and for which it does not have permissions to write, hence the failure.

Adding a "-" to the runuser in the command solves the problem, ie changing

runuser -s /bin/sh qpidd -c "/usr/sbin/$prog --check > $pidfile"

to

runuser - -s /bin/sh qpidd -c "/usr/sbin/$prog --check > $pidfile"

(thanks, Gordon!)

Fixed in r.953687

[Upstream jira QPID-2666]