Red Hat Bugzilla – Bug 589683
Broker misconfiguration causes broker stop to fail
Last modified: 2011-07-15 15:15:20 EDT
If the broker configuration file /etc/qpidd.conf is misconfigured, the broker will start, but it cannot be stopped again (using service stop).
Starting with clean /var/lib/qpidd and /var/run/qpidd dirs:
[root@mrg10 ~]# cat /etc/qpidd.conf
[root@mrg10 ~]# service qpidd start
Starting Qpid AMQP daemon: 2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2010-05-06 12:50:27 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
[ OK ]
Unexpected error: Can't open log file: test.log
[root@mrg10 ~]# service qpidd stop
Stopping Qpid AMQP daemon: [FAILED]
[root@mrg10 ~]# ps -eaf | grep qpidd
qpidd 4870 1 0 12:50 ? 00:00:00 /usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
root 4899 3322 0 12:52 pts/0 00:00:00 grep qpidd
NOTE 1: In spite of the broker error above, the log file is in fact created in the defualt directory (/var/lib/qpidd/test.log).
NOTE 2: Changing the config file to an absolute path (log-to-file=/var/lib/qpidd/test.log) fixes the broker startup error; in this case the service stop works ok.
NOTE 3: Both these cases place a pid file /var/run/qpidd/qpidd.5672.pid which contains a binary PID number. However, if the config file is fixed as in NOTE 2, then in addition a second pid file is created: /var/run/qpidd.pid which contains a string representation of the PID. It is likely this second file which causes the service stop to fail.
Tested on RHEL-6 using qpid-cpp-server-0.7.935473-1.el6.x86_64 and its dependents.
This is a much narrower, but more complex case than I initially thought.
The use of the "log-to-file" option in the config file is the source of this error. Genuine configuration file errors are either ignored or result in a failure to start the daemon in the first place.
When the broker daemon is started, it reads the config file and opens the log file as expected. This also creates an empty pid file. The startup script then uses "qpidd --check" to get the PID and write it into the empty pid file. This instance of qpidd reads the same config file and then attempts to open the same log file as the broker itself, causing it to fail with the message "Unexpected error: Can't open log file: test.log" (in the example above). This failure in turn leaves the pid file empty, and hence the use of "service stop" fails.
There is no simple solution to this that is not "hacky".
Ideally, this should be solved within qpidd itself - for instance if the daemon service options --quit or --check are used, then logging and module loading should be disabled.
Ignore analysis in comment #2 above - incorrect conclusion.
Absolute paths work ok. It turns out that using log-to-file without an absolute path causes the "qpidd --check" to run in the incorrect directory and for which it does not have permissions to write, hence the failure.
Adding a "-" to the runuser in the command solves the problem, ie changing
runuser -s /bin/sh qpidd -c "/usr/sbin/$prog --check > $pidfile"
runuser - -s /bin/sh qpidd -c "/usr/sbin/$prog --check > $pidfile"
Fixed in r.953687
[Upstream jira QPID-2666]