Bug 1315873

Summary:	Improve logging for exchange not found errors
Product:	Red Hat OpenStack	Reporter:	John Eckersberg <jeckersb>
Component:	python-oslo-messaging	Assignee:	John Eckersberg <jeckersb>
Status:	CLOSED ERRATA	QA Contact:	Asaf Hirshberg <ahirshbe>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	apevec, lhh, srevivo, vstinner
Target Milestone:	async	Keywords:	ZStream
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-oslo-messaging-1.8.3-7.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-01-19 13:27:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description John Eckersberg 2016-03-08 20:32:55 UTC

In Kilo (OSP7) prior, oslo.messaging logs a verbose (and scary looking) backtrace when a reply exchange is missing.  Moreso, for each message received with missing reply exchange, the code will retry sending for 60 seconds with 0.25s sleep intervals.  This means that when the original client really went away, 240 backtraces (60s * 4 tries/s) will be logged per message.  This (1) creates a lot of unnecessary I/O, (2) uses a lot of disk space, and (3) misleads (and terrifies) operators into believing severe degredation of service is in progress (maybe, maybe not).

This was corrected as part of Liberty (OSP8) here:

https://review.openstack.org/#/c/193037/

But that change is unable to be backported to Kilo because it introduces new dependencies (cachetools).  Here's the rejected Kilo backport:

https://review.openstack.org/#/c/193484/

We should backport just the logging portion of that change.

Comment 1 John Eckersberg 2016-10-10 20:51:37 UTC

https://code.engineering.redhat.com/gerrit/#/c/85299/2

Comment 2 John Eckersberg 2016-12-16 21:04:57 UTC

python-oslo-messaging-1.8.3-7.el7ost - https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=528968

Comment 4 Asaf Hirshberg 2017-01-03 07:46:44 UTC

Code verified on rhos-7.
[stack@undercloud-0 ~]$ rpm -qa |grep python-oslo-messaging
python-oslo-messaging-1.8.3-7.el7ost.noarch

Comment 6 errata-xmlrpc 2017-01-19 13:27:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0158.html