Bug 1280019

Summary: Failure to reconnect when connection lost during message acknowlegement
Product: Red Hat OpenStack Reporter: Flavio Percoco <fpercoco>
Component: openstack-ceilometerAssignee: Mehdi ABAAKOUK <mabaakou>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.0 (RHEL 7)CC: apevec, dmaley, eglynn, fpercoco, jeckersb, jruzicka, jtaleric, kgiusti, lhh, mwagner, oblaut, sclewis, yeylon
Target Milestone: asyncKeywords: GSSTriaged, Triaged, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ceilometer-2014.1.5-3.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1278177
: 1300044 (view as bug list) Environment:
Last Closed: 2016-02-11 15:02:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1278177    
Bug Blocks: 1278176, 1300044    

Comment 5 Yurii Prokulevych 2016-02-09 15:52:04 UTC 
Verified in ceilometer*-2014.1.5-3.el7ost.noarch

Excerpt from collector.log:
2016-02-09 15:30:18.754 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to process message ... skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 170, in _callback_handler
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     callback(msg)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 436, in __call__
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     self.msg_id_cache.check_duplicate_message(message_data)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 335, in check_duplicate_message
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common DuplicateMessageError: Found duplicate message(243cedb2ae9442b6bca105d1fba070b0). Skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.754 89594 DEBUG ceilometer.openstack.common.rpc.common [-] Yurii.ConnectionForcibyClosed. RHBZ:1280019 _callback_handler /usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_
kombu.py:178
2016-02-09 15:30:18.755 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 615, in ensure
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return method(*args, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 702, in _consume
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 325, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return amqp_method(channel, args, content)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1908, in _basic_deliver
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     fun(msg)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 213, in _callback
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._callback_handler(message, callback)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 179, in _callback_handler
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     message.ack()
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/message.py", line 88, in ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel.basic_ack(self.delivery_tag)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1583, in basic_ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._send_method((60, 80), args)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel_id, method_sig, args, content,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     write_frame(1, channel, payload)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     frame_type, channel, size, payload, 0xce,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 310, in sendall
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     tail = self.send(data, flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     total_sent += fd.send(data[total_sent:], flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common error: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.34.75.91:5672
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 1.0 seconds...
2016-02-09 15:30:19.780 89594 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.34.75.91:5672
Comment 7 errata-xmlrpc 2016-02-11 15:02:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0169.html