While working with a partner on some problems in their system, I have observed two instances where the qpid client library gets into a bad state and the qpid connection thread in nova never recovers. An example of the exception is: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func return infunc(*args, **kwargs) File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread self.consume() File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume it.next() File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume yield self.ensure(_error_callback, _consume) File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure return method(*args, **kwargs) File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume nxt_receiver = self.session.next_receiver(timeout=timeout) File "<string>", line 6, in next_receiver File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver if self._ecwait(lambda: self.incoming, timeout): File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait result = self._ewait(lambda: self.closed or predicate(), timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait result = self.connection._ewait(lambda: self.error or predicate(), timeout) File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait self.check_error() File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error raise self.error InternalError: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 667, in write self._op_dec.write(*self._seg_dec.read()) File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write if self.op.headers is None: AttributeError: 'NoneType' object has no attribute 'headers' There is some code that automatically detects if the thread dies with an exception. It will sleep for a second a retry. The code will sit in this loop forever. Every time it tries to run again it will hit this error immediately. As a result, you see a message like this every minute or so: 2014-04-06 09:03:49.014 125211 ERROR root [-] Unexpected exception occurred 60 time(s)... retrying. Part of the issue is that I don't think this should ever happen. However, if it does, Nova should be more tolerant and reset the connection instead being stuck in this error for forever.
This patch has been merged into both oslo.messaging and the rpc library in oslo-incubator. In RHOS 4.0, nothing had been converted to oslo.messaging, so this fix needs to be backported to all of the projects that include rpc from oslo-incubator. I will be cloning this bug to all affected projects.
*** Bug 1098827 has been marked as a duplicate of this bug. ***
*** Bug 1108959 has been marked as a duplicate of this bug. ***
How do we repro this condition? Is there a way to determine which of the qpid child threads is the connection thread? If so, I could try to kill the connection thread, and see if the problem persists. Actually, as I am writing this, I see Russel posted this as an internal error in the python-qpid library. Is there a version of python-qpid with a fix for this?
(In reply to Sean Toner from comment #6) > How do we repro this condition? Is there a way to determine which of the > qpid child threads is the connection thread? If so, I could try to kill the > connection thread, and see if the problem persists. Reproducing this will be very difficult. I honestly wouldn't bother. Time is better spent elsewhere. > Actually, as I am writing this, I see Russel posted this as an internal > error in the python-qpid library. Is there a version of python-qpid with a > fix for this? Yes, this was triggered when we hit this bug: https://issues.apache.org/jira/browse/QPID-5700 https://bugzilla.redhat.com/show_bug.cgi?id=1088004 The fix was in python-qpid-0.18-10.el7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-1084.html