Bug 1052724

Summary: openstack - qpidd service not enabled - excessive errors in journal after reboot
Product: [Fedora] Fedora Reporter: John Ellson <john.ellson>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: ichavero, itamar, Jan.van.Eldik, john.ellson, mmagr, pbrady, p
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-20 14:52:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/tmp/packstack/20140113-131402-1fhdVa/manifests/135.207.139.248_qpid.pp.log none

Description John Ellson 2014-01-13 22:47:56 UTC 
Description of problem:
qpidd service not enabled - excessive errors in journal after reboot

Version-Release number of selected component (if applicable):
openstack-packstack-2013.2.1-0.25.dev936.fc20.noarch

How reproducible:


Steps to Reproduce:
1.reboot
2.joutnalctl -f | grep ERROR
3.

Actual results:
Jan 13 17:37:36 work ceilometer-agent-compute[1174]: 2014-01-13 17:37:36.629 1174 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:39 work ceilometer-collector[1248]: 2014-01-13 17:37:39.078 1248 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:39 work ceilometer-agent-central[1153]: 2014-01-13 17:37:39.240 1153 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:39 work ceilometer-alarm-notifier[1247]: 2014-01-13 17:37:39.261 1247 ERROR ceilometer.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:42 work neutron-l3-agent[3079]: 2014-01-13 17:37:42.257 3079 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:42 work neutron-l3-agent[3079]: 2014-01-13 17:37:42.271 3079 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:44 work neutron-openvswitch-agent[3082]: 2014-01-13 17:37:44.926 3082 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:44 work neutron-dhcp-agent[3081]: 2014-01-13 17:37:44.933 3081 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds
Jan 13 17:37:44 work neutron-dhcp-agent[3081]: 2014-01-13 17:37:44.934 3081 ERROR neutron.openstack.common.rpc.impl_qpid [-] Unable to connect to AMQP server: [Errno 111] ECONNREFUSED. Sleeping 60 seconds


Expected results:
1. qpidd service should be enabled by openstack-packstack

2. openstack should just give up in the presence of errors like this, and not overload the logging system

Additional info:
Comment 1 Martin Magr 2014-01-16 15:55:16 UTC 
By any chance aren't those message appearing only until qpidd starts? Since there is no service synchronization it can happen that ceilometer/neutron/... services will start before qpidd. 

From source code PoV qpidd service is enabled.
Comment 2 John Ellson 2014-01-16 18:42:39 UTC 
"aren't those message appearing only until qpidd starts?"   Yes, but qpidd wasn't getting started after reboot because it wasn't enabled.

"Since there is no service synchronization..."  Really?  Isn't this facility provided by systemd?  

BTW.   Is there (or could there be) a single systemd target that starts all these 100s of services?  (And stops them).
Comment 3 Pádraig Brady 2014-01-17 05:14:49 UTC 
Note the openstack-service command which you might find useful:

  openstack-service start openstack*
Comment 4 Martin Magr 2014-01-17 11:06:27 UTC 
Ok, if it wasn't enabled, could you please upload packstack log from the installation? You will find it in /var/tmp/packstack/<timestamp>-<hash>/manifests/<IP>_qpidd.pp.log. That way I can check if the qpidd was enabled by Puppet or not and search for some potential errors. Thanks in advance
Comment 5 John Ellson 2014-01-17 14:16:20 UTC 
Created attachment 851641 [details]
/var/tmp/packstack/20140113-131402-1fhdVa/manifests/135.207.139.248_qpid.pp.log
Comment 6 Martin Magr 2014-01-17 14:37:43 UTC 
The log does not contain notice from installing and enabling qpid, so either this is log from the second run of packstack or qpid was installed and enabled before. Can you check /var/tmp/packstack and check if there is oldest logs (eg. I would need the log from first run).
Comment 7 John Ellson 2014-01-17 19:17:52 UTC 
I had no older logs ..

I cleaned everything out and did a  "packstack --allinone"
again, successfully, but I was unable to recreate the problem.

Now my logs are mostly getting filled with nova-compute and nova-conductor DEBUG messages.   Can they be turned off?

e.g.
Jan 17 14:05:16 work nova-conductor[11276]: 2014-01-17 14:05:16.035 11276 DEBUG qpid.messaging.io.ops [-] RCVD[4af0248]: SessionCommandPoint(command_id=serial(0), command_offset=0) write /usr/lib/python2.7/site-packages/qpid/messaging/driver.py:642

I tried rebooting the system, but it came back with all services running, AFAICT.
Comment 8 Martin Magr 2014-01-20 14:52:14 UTC 
In case you will hit the same problem in future please reopen this bug or create a new one.