Bug 1050216 – Thread consuming qpid messages can die silently

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1050216 - Thread consuming qpid messages can die silently

Summary:	Thread consuming qpid messages can die silently

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-heat (Show other bugs)
Sub Component:
Version:	3.0
Hardware:	Unspecified Unspecified

Priority	urgent Severity urgent
Target Milestone:	z4
Target Release:	3.0
Assigned To:	Jeff Peeler
QA Contact:	Amit Ugol
Docs Contact:

URL:
Whiteboard:
Keywords:	OtherQA, ZStream

Depends On:	1050213 1050217
Blocks:	1050214 1050215
	Show dependency tree / graph

Reported:	2014-01-08 16:05 EST by Russell Bryant
Modified:	2014-01-30 14:51 EST (History)
CC List:	12 users (show)

See Also:
Fixed In Version:	openstack-heat-2013.1.4-2.el6ost
Doc Type:	Bug Fix
Doc Text:	Cause: All potential errors were not explicitly handled in the QPID driver's consuming thread. Consequence: Unhandled errors encountered by the QPID driver's consuming thread could cause it to die silently, so that no further incoming messages are received by that agent. Fix: The consuming thread has been made more resilient to potential errors. Result: Errors are now logged and the message consumption logic is then retried.
Story Points:	---
Clone Of:	1050213
Environment:
Last Closed:	2014-01-30 14:51:07 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1189711	None	None	None	Never
Red Hat Product Errata	RHBA-2014:0110	normal	SHIPPED_LIVE	Red Hat Enterprise Linux OpenStack Platform 3 bug fix advisory	2014-01-30 19:48:26 EST

Groups: None (edit)

Description Russell Bryant 2014-01-08 16:05:15 EST

+++ This bug was initially created as a clone of Bug #1050213 +++

The code for receiving and processing qpid messages runs in its own greenthread.  Unfortunately, there is a code path that if it raises an exception, the greenthread will die silently, without any entry in the log file.

In particular, the code in question is:

https://git.openstack.org/cgit/openstack/nova/tree/nova/openstack/common/rpc/impl_qpid.py?h=stable/grizzly#n468

if self.session.next_receiver() raises an exception other than qpid's Empty or ConnectionError exceptions, the thread will die and no more messages will be received.

The fix is to backport the portion of the following change that applies to impl_qpid.  That includes the new decorator in excutils and the change to impl_qpid.py.

Comment 1 Russell Bryant 2014-01-08 16:12:56 EST

The change that needs to be backported is: https://review.openstack.org/#/c/32235/13

I also attached a version of the backport for nova to bug 1050213

Comment 3 Jeff Peeler 2014-01-09 13:00:36 EST

Heat actually already contains these changes as Oslo was updated in commit 00e71c3d7810e2f9e12f57c0a713d2b8939bb70f. (Part of the fix was previously updated in commit eca7fdae8c61cce8b9be8d7c4d95664ff2c02694.)

Comment 4 Lon Hohberger 2014-01-14 14:46:54 EST

This is a bug against RHOS 3.0 Grizzly, not Havana.

Those commits are not in the RHOS 3.0 heat packages nor upstream stable/grizzly.

Comment 6 Amit Ugol 2014-01-27 10:19:26 EST

Changes from https://review.openstack.org/#/c/32235/13 were integrated to:
...heat/openstack/common/excutils.py
...heat/openstack/common/rpc/impl_kombu.py
...heat/openstack/common/rpc/impl_qpid.py

os:     RHEL 6.5
puddle: OpenStack/Grizzly/2014-01-22.1

Comment 8 errata-xmlrpc 2014-01-30 14:51:07 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0110.html

Note You need to log in before you can comment on or make changes to this bug.