Bug 1312912 - Option "rabbit_retry_backoff" from component config has no effect when autoretrying connection
Option "rabbit_retry_backoff" from component config has no effect when autore...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-oslo-messaging (Show other bugs)
8.0 (Liberty)
Unspecified Unspecified
unspecified Severity medium
: ga
: 8.0 (Liberty)
Assigned To: Flavio Percoco
Leonid Natapov
:
Depends On:
Blocks: 1313522
  Show dependency treegraph
 
Reported: 2016-02-29 09:03 EST by Marian Krcmarik
Modified: 2016-04-07 17:31 EDT (History)
6 users (show)

See Also:
Fixed In Version: python-oslo-messaging-2.5.0-5.el7ost
Doc Type: Bug Fix
Doc Text:
When the RabbitMQ service fails to deliver an AMQP message from one OpenStack service to another, it reconnects and retries delivery. The "rabbit_retry_backoff" option, whose default is 2 seconds, is supposed to control the pace of retries; however, retries were previously done every second irrespective of the configured value of this option. The consequence of this problem was excessive retries, for example, when an endpoint was not available. This problem has now been fixed, and the "rabbit_retry_backoff" option, as explicitly configured or with the default value of two seconds, properly controls message delivery retries.
Story Points: ---
Clone Of:
: 1313522 (view as bug list)
Environment:
Last Closed: 2016-04-07 17:31:44 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Marian Krcmarik 2016-02-29 09:03:24 EST
Description of problem:
There is a messed up code in _drivers/impl_rabbit.py which causing that autoretry reconnection is always triggered every second despite rabbit_retry_backoff option being set (in default to 2).

Specifically:
865             autoretry_method = self.connection.autoretry(
866                 execute_method, channel=self.channel,
867                 max_retries=retry,
868                 errback=on_error,
869                 interval_start=self.interval_start or 1,
870                 interval_step=self.interval_stepping,
871                 on_revive=on_reconnection,
872             )
There is no interval_max=self.interval_max parameter specified (hardcoded to value 30).
Method kombu.connection.autoretry() calls kombu.connectio.ensure() which sets interval_max to 1 by default so backing off cannot have any effect since max interval is 1.

It would be nice to be able to set interval_max in component configs and not hardcoded it to 30 maybe.

Version-Release number of selected component (if applicable):
python-oslo-messaging-2.5.0-4.el7ost.noarch

How reproducible:
Always

Steps to Reproduce:
1. Set rabbit_retry_backoff to anything 2+
2. Take down rabbitmq server

Actual results:
Reconenction is actually happening every second, instead of interval being backed off by 2+ seconds each try up to 30 seconds

Expected results:


Additional info:
Comment 2 Victor Stinner 2016-03-17 10:10:11 EDT
python-oslo-messaging-2.5.0-5.el7ost is ready for tests.
Comment 5 errata-xmlrpc 2016-04-07 17:31:44 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html

Note You need to log in before you can comment on or make changes to this bug.