Bug 1933491 - [RHOSP 16.1] aodh throwing errors when haproxy closes TCP connections to Redis
Summary: [RHOSP 16.1] aodh throwing errors when haproxy closes TCP connections to Redis
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 16.1 (Train)
Hardware: Unspecified
OS: Linux
urgent
high
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Emma Foley
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-28 16:18 UTC by Matthias Runge
Modified: 2022-12-07 20:25 UTC (History)
16 users (show)

Fixed In Version: puppet-tripleo-11.5.0-1.20220331172929.f716ef5.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 1924373
Environment:
Last Closed: 2022-12-07 20:24:45 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 788259 0 None MERGED Retry on redis connection errors 2022-09-26 11:43:17 UTC
Red Hat Issue Tracker OSP-1610 0 None None None 2021-11-18 14:20:31 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:25:37 UTC

Comment 2 Matthias Runge 2021-02-28 16:47:08 UTC 
After a second look, I think this should be good in tooz, but may require additional fixes elsewhere. The exception is pushed up through the stack.
Comment 6 Matthias Runge 2021-03-03 08:05:27 UTC 
2021-03-03 00:02:12.415 24 ERROR aodh.coordination Traceback (most recent call last):
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination OSError: Connection closed by server.
2021-03-03 00:02:12.415 24 ERROR aodh.coordination 
2021-03-03 00:02:12.415 24 ERROR aodh.coordination During handling of the above exception, another exception occurred:
2021-03-03 00:02:12.415 24 ERROR aodh.coordination 
2021-03-03 00:02:12.415 24 ERROR aodh.coordination Traceback (most recent call last):
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 47, in _translate_failures
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     yield
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 521, in heartbeat
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     value=self.STILL_ALIVE)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 1391, in psetex
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     return self.execute_command('PSETEX', name, time_ms, value)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 775, in execute_command
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     return self.parse_response(connection, command_name, **options)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 789, in parse_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = connection.read_response()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     raise e
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = self._parser.read_response()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = self._buffer.readline()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     self._read_from_socket()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     (e.args,))
2021-03-03 00:02:12.415 24 ERROR aodh.coordination redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination
Comment 9 Matthias Runge 2021-03-05 13:29:52 UTC 
Fix proposed to https://review.opendev.org/c/openstack/tooz/+/778878
the gate is broken there, as it seems.
Comment 14 Matthias Runge 2021-06-10 15:11:13 UTC 
There is also https://review.opendev.org/c/openstack/tooz/+/795791 to merge for this upstream and then to backport it.
Comment 15 Matthias Runge 2021-07-23 06:42:17 UTC 
This is still in upstream review.
Comment 22 Emma Foley 2022-06-22 14:03:21 UTC 
There is a backport to Ussuri in progress upstream: https://review.opendev.org/c/openstack/tooz/+/847152

Once the Ussuri backport is merged, the train one can be proposed.
Comment 23 Emma Foley 2022-07-19 16:24:38 UTC 
The patch has been backported to train: https://review.opendev.org/c/openstack/tooz/+/848654
Comment 25 Martin Magr 2022-09-07 16:44:16 UTC 
This issue has the same cause and fix as BZ1924373
Comment 29 Leonid Natapov 2022-11-06 04:24:22 UTC 
No errors or traces appear in ceilometer logs.
Comment 35 errata-xmlrpc 2022-12-07 20:24:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795
Note You need to log in before you can comment on or make changes to this bug.