Description of problem: qpidd service not enabled - excessive errors in journal after reboot Version-Release number of selected component (if applicable): openstack-packstack-2013.2.1-0.25.dev936.fc20.noarch How reproducible: Steps to Reproduce: 1.reboot 2.joutnalctl -f | grep ERROR 3. Actual results: Jan 13 17:37:36 work ceilometer-agent-compute[1174]: 2014-01-13 17:37:36.629 1174 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:39 work ceilometer-collector[1248]: 2014-01-13 17:37:39.078 1248 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:39 work ceilometer-agent-central[1153]: 2014-01-13 17:37:39.240 1153 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:39 work ceilometer-alarm-notifier[1247]: 2014-01-13 17:37:39.261 1247 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:42 work neutron-l3-agent[3079]: 2014-01-13 17:37:42.257 3079 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:42 work neutron-l3-agent[3079]: 2014-01-13 17:37:42.271 3079 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:44 work neutron-openvswitch-agent[3082]: 2014-01-13 17:37:44.926 3082 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:44 work neutron-dhcp-agent[3081]: 2014-01-13 17:37:44.933 3081 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Jan 13 17:37:44 work neutron-dhcp-agent[3081]: 2014-01-13 17:37:44.934 3081 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds Expected results: 1. qpidd service should be enabled by openstack-packstack 2. openstack should just give up in the presence of errors like this, and not overload the logging system Additional info:
By any chance aren't those message appearing only until qpidd starts? Since there is no service synchronization it can happen that ceilometer/neutron/... services will start before qpidd. From source code PoV qpidd service is enabled.
"aren't those message appearing only until qpidd starts?" Yes, but qpidd wasn't getting started after reboot because it wasn't enabled. "Since there is no service synchronization..." Really? Isn't this facility provided by systemd? BTW. Is there (or could there be) a single systemd target that starts all these 100s of services? (And stops them).
Note the openstack-service command which you might find useful: openstack-service start openstack*
Ok, if it wasn't enabled, could you please upload packstack log from the installation? You will find it in /var/tmp/packstack/<timestamp>-<hash>/manifests/<IP>_qpidd.pp.log. That way I can check if the qpidd was enabled by Puppet or not and search for some potential errors. Thanks in advance
Created attachment 851641 [details] /var/tmp/packstack/20140113-131402-1fhdVa/manifests/135.207.139.248_qpid.pp.log
The log does not contain notice from installing and enabling qpid, so either this is log from the second run of packstack or qpid was installed and enabled before. Can you check /var/tmp/packstack and check if there is oldest logs (eg. I would need the log from first run).
I had no older logs .. I cleaned everything out and did a "packstack --allinone" again, successfully, but I was unable to recreate the problem. Now my logs are mostly getting filled with nova-compute and nova-conductor DEBUG messages. Can they be turned off? e.g. Jan 17 14:05:16 work nova-conductor[11276]: 2014-01-17 14:05:16.035 11276 DEBUG qpid.messaging.io.ops [-] RCVD[4af0248]: SessionCommandPoint(command_id=serial(0), command_offset=0) write /usr/lib/python2.7/site-packages/qpid/messaging/driver.py:642 I tried rebooting the system, but it came back with all services running, AFAICT.
In case you will hit the same problem in future please reopen this bug or create a new one.