Bug 1280019 - Failure to reconnect when connection lost during message acknowlegement
Failure to reconnect when connection lost during message acknowlegement
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-ceilometer (Show other bugs)
5.0 (RHEL 7)
Unspecified Unspecified
urgent Severity urgent
: async
: 5.0 (RHEL 7)
Assigned To: Mehdi ABAAKOUK
Yurii Prokulevych
: GSSTriaged, Triaged, ZStream
Depends On: 1278177
Blocks: 1278176 1300044
  Show dependency treegraph
 
Reported: 2015-11-10 13:14 EST by Flavio Percoco
Modified: 2016-02-11 10:02 EST (History)
14 users (show)

See Also:
Fixed In Version: openstack-ceilometer-2014.1.5-3.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1278177
: 1300044 (view as bug list)
Environment:
Last Closed: 2016-02-11 10:02:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 5 Yurii Prokulevych 2016-02-09 10:52:04 EST
Verified in ceilometer*-2014.1.5-3.el7ost.noarch

Excerpt from collector.log:
2016-02-09 15:30:18.754 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to process message ... skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 170, in _callback_handler
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     callback(msg)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 436, in __call__
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     self.msg_id_cache.check_duplicate_message(message_data)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/amqp.py", line 335, in check_duplicate_message
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common DuplicateMessageError: Found duplicate message(243cedb2ae9442b6bca105d1fba070b0). Skipping it.
2016-02-09 15:30:18.754 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.754 89594 DEBUG ceilometer.openstack.common.rpc.common [-] Yurii.ConnectionForcibyClosed. RHBZ:1280019 _callback_handler /usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_
kombu.py:178
2016-02-09 15:30:18.755 89594 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 615, in ensure
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return method(*args, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 702, in _consume
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.connection.drain_events(timeout=timeout)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return self.transport.drain_events(self.connection, **kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return connection.drain_events(**kwargs)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 325, in drain_events
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     return amqp_method(channel, args, content)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1908, in _basic_deliver
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     fun(msg)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 213, in _callback
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._callback_handler(message, callback)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 179, in _callback_handler
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     message.ack()
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/kombu/message.py", line 88, in ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel.basic_ack(self.delivery_tag)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1583, in basic_ack
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self._send_method((60, 80), args)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     self.channel_id, method_sig, args, content,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/method_framing.py", line 221, in write_method
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     write_frame(1, channel, payload)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 182, in write_frame
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     frame_type, channel, size, payload, 0xce,
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 310, in sendall
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     tail = self.send(data, flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common     total_sent += fd.send(data[total_sent:], flags)
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common error: [Errno 32] Broken pipe
2016-02-09 15:30:18.755 89594 TRACE ceilometer.openstack.common.rpc.common
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 10.34.75.91:5672
2016-02-09 15:30:18.756 89594 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 1.0 seconds...
2016-02-09 15:30:19.780 89594 INFO ceilometer.openstack.common.rpc.common [-] Connected to AMQP server on 10.34.75.91:5672
Comment 7 errata-xmlrpc 2016-02-11 10:02:59 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0169.html

Note You need to log in before you can comment on or make changes to this bug.