Bug 2100879 - From time to time memcached stops processing requests and brings down OpenStack control plane
Summary: From time to time memcached stops processing requests and brings down OpenSta...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-dogpile-cache
Version: 16.1 (Train)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Hervé Beraud
QA Contact: nlevinki
URL:
Whiteboard:
Depends On: 1893205
Blocks: 2046185 2101864 2101865
TreeView+ depends on / blocked
 
Reported: 2022-06-24 14:32 UTC by David Hill
Modified: 2022-12-08 21:21 UTC (History)
36 users (show)

Fixed In Version: python-dogpile-cache-0.6.8-3.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, dogpile.cache support for dead_retry and socket_timeout was not implemented for the memcached back end. The oslo.cache mechanism filled the arguments dictionary with values for dead_retry and socket_timeout, but dogpile.cache ignored the values so the defaults of 30s for dead_retry and 3s for socket_timeout were used. When using dogpile.cache.memcached as the cache back end on the Identity service (keystone), and then taking down one of the memcached instances, the memcache server objects set their deaduntil value to 30 seconds in the future. When a request came in to an API server with two memcached servers configured, one of which was unroutable, it took approximately 15 seconds for it to try each of those servers in each thread it created and reach the three-second socket timeout limit every time it encountered the one that was down. By the time the user issued another request, the deaduntil value was reached and the whole cycle was repeated. With this update, dogpile.cache consumes dead_retry and socket_timeout arguments passed by oslo.cache.
Clone Of: 1893205
: 2101864 (view as bug list)
Environment:
Last Closed: 2022-12-07 20:27:07 UTC
Target Upstream Version:
Embargoed:
hberaud: needinfo+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github sqlalchemy/dogpile.cache/commit/1de93aab14c1274f20c1f44f8adff3b143c864f6 0 None None None 2022-06-28 13:05:49 UTC
Red Hat Issue Tracker OSP-15982 0 None None None 2022-06-24 14:36:10 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:27:52 UTC

Comment 27 errata-xmlrpc 2022-12-07 20:27:07 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795
Note You need to log in before you can comment on or make changes to this bug.