Bug 1300044

Summary: Failure to reconnect when connection lost during message acknowlegement
Product: Red Hat OpenStack Reporter: Lon Hohberger <lhh>
Component: openstack-ceilometerAssignee: Eoghan Glynn <eglynn>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.0 (RHEL 7)CC: apevec, dmaley, eglynn, fpercoco, jeckersb, jruzicka, jtaleric, kgiusti, lhh, mabaakou, mwagner, oblaut, sclewis, yeylon, yprokule
Target Milestone: asyncKeywords: GSSTriaged, Triaged, ZStream
Target Release: 5.0 (RHEL 6)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ceilometer-2014.1.5-3.el6ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1280019 Environment:
Last Closed: 2016-03-23 14:27:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1278177, 1280019    
Bug Blocks: 1278176    

Comment 2 Yurii Prokulevych 2016-03-22 10:28:44 UTC 
Verified in openstack-ceilometer*-2014.1.5-3.el6ost.noarch

Excerpt from ceilometer/collector.log:2016-03-22 11:21:54.845 85711 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 32] Broken pipe
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 616, in ensure
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     return method(*args, **kwargs)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 703, in _consume
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 281, in drain_events
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqp.py", line 94, in drain_events
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/connection.py", line 288, in drain_events
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     return amqp_method(channel, args, content)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 1886, in _basic_deliver
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     fun(msg)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 213, in _callback
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     self._callback_handler(message, callback)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 189, in _callback_handler
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     message.ack()
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/kombu/transport/base.py", line 95, in ack
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     self.channel.basic_ack(self.delivery_tag)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 1565, in basic_ack
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     self._send_method((60, 80), args)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 58, in _send_method
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     self.channel_id, method_sig, args, content,
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 224, in write_method
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     write_frame(1, channel, payload)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 160, in write_frame
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     pack('>BHI%dsB' % size, frame_type, channel, size, payload, 0xce),
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 310, in sendall
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     tail = self.send(data, flags)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common     total_sent += fd.send(data[total_sent:], flags)
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common error: [Errno 32] Broken pipe
2016-03-22 11:21:54.845 85711 TRACE ceilometer.openstack.common.rpc.common
2016-03-22 11:21:54.846 85711 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.34.75.91:5672
2016-03-22 11:21:54.847 85711 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 1.0 seconds...
2016-03-22 11:21:55.852 85711 DEBUG ceilometer.dispatcher.database [req-70e6c83a-136f-44bc-976a-b3215b9d6116 - - - - -] metering data volume for bb41ee2a-f591-46c7-84ef-0d50814ce2cf @ 2016-03-22 10:21:54.829709:
 1 record_metering_data /usr/lib/python2.6/site-packages/ceilometer/dispatcher/database.py:56
2016-03-22 11:21:55.859 85711 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.34.75.91:5672
Comment 4 errata-xmlrpc 2016-03-23 14:27:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0498.html