Bug 1280019 - Failure to reconnect when connection lost during message acknowlegement
Summary: Failure to reconnect when connection lost during message acknowlegement
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: async
: 5.0 (RHEL 7)
Assignee: Mehdi ABAAKOUK
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On: 1278177
Blocks: 1278176 1300044
TreeView+ depends on / blocked
 
Reported: 2015-11-10 18:14 UTC by Flavio Percoco
Modified: 2023-02-22 23:02 UTC (History)
13 users (show)

Fixed In Version: openstack-ceilometer-2014.1.5-3.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of: 1278177
: 1300044 (view as bug list)
Environment:
Last Closed: 2016-02-11 15:02:59 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0169 0 normal SHIPPED_LIVE openstack-ceilometer bug fix advisory 2016-02-11 20:02:25 UTC

Comment 5 Yurii Prokulevych 2016-02-09 15:52:04 UTC 
Verified in ceilometer*-2014.1.5-3.el7ost.noarch

Excerpt from collector.log:
2016-02-09 15:30:18.754 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to process message ... skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 170, in _callback_handler
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     callback(msg)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 436, in __call__
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     self.msg_id_cache.check_duplicate_message(message_data)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 335, in check_duplicate_message
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common DuplicateMessageError: Found duplicate message(243cedb2ae9442b6bca105d1fba070b0). Skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.754 89594 DEBUG ceilometer.openstack.common.rpc.common [-] Yurii.ConnectionForcibyClosed. RHBZ:1280019 _callback_handler /usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_
kombu.py:178
2016-02-09 15:30:18.755 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 615, in ensure
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return method(*args, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 702, in _consume
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 325, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return amqp_method(channel, args, content)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1908, in _basic_deliver
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     fun(msg)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 213, in _callback
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._callback_handler(message, callback)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 179, in _callback_handler
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     message.ack()
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/message.py", line 88, in ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel.basic_ack(self.delivery_tag)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1583, in basic_ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._send_method((60, 80), args)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel_id, method_sig, args, content,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     write_frame(1, channel, payload)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     frame_type, channel, size, payload, 0xce,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 310, in sendall
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     tail = self.send(data, flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     total_sent += fd.send(data[total_sent:], flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common error: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.34.75.91:5672
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 1.0 seconds...
2016-02-09 15:30:19.780 89594 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.34.75.91:5672
Comment 7 errata-xmlrpc 2016-02-11 15:02:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0169.html
Note You need to log in before you can comment on or make changes to this bug.