Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1933491

Summary:	[RHOSP 16.1] aodh throwing errors when haproxy closes TCP connections to Redis
Product:	Red Hat OpenStack	Reporter:	Matthias Runge <mrunge>
Component:	puppet-tripleo	Assignee:	Emma Foley <efoley>
Status:	CLOSED ERRATA	QA Contact:	Leonid Natapov <lnatapov>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	16.1 (Train)	CC:	apevec, bperkins, eduen, efoley, fgadkano, jjoyce, jschluet, lhh, lmadsen, mmagr, mrunge, mvalsecc, nlevinki, slinaber, tkajinam, tvignaud
Target Milestone:	z9	Keywords:	Triaged, ZStream
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	puppet-tripleo-11.5.0-1.20220331172929.f716ef5.el8ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	1924373	Environment:
Last Closed:	2022-12-07 20:24:45 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 2 Matthias Runge 2021-02-28 16:47:08 UTC

After a second look, I think this should be good in tooz, but may require additional fixes elsewhere. The exception is pushed up through the stack.

Comment 6 Matthias Runge 2021-03-03 08:05:27 UTC

2021-03-03 00:02:12.415 24 ERROR aodh.coordination Traceback (most recent call last):
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination OSError: Connection closed by server.
2021-03-03 00:02:12.415 24 ERROR aodh.coordination 
2021-03-03 00:02:12.415 24 ERROR aodh.coordination During handling of the above exception, another exception occurred:
2021-03-03 00:02:12.415 24 ERROR aodh.coordination 
2021-03-03 00:02:12.415 24 ERROR aodh.coordination Traceback (most recent call last):
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 47, in _translate_failures
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     yield
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 521, in heartbeat
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     value=self.STILL_ALIVE)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 1391, in psetex
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     return self.execute_command('PSETEX', name, time_ms, value)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 775, in execute_command
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     return self.parse_response(connection, command_name, **options)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/client.py", line 789, in parse_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = connection.read_response()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     raise e
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = self._parser.read_response()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     response = self._buffer.readline()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     self._read_from_socket()
2021-03-03 00:02:12.415 24 ERROR aodh.coordination   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
2021-03-03 00:02:12.415 24 ERROR aodh.coordination     (e.args,))
2021-03-03 00:02:12.415 24 ERROR aodh.coordination redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2021-03-03 00:02:12.415 24 ERROR aodh.coordination

Comment 9 Matthias Runge 2021-03-05 13:29:52 UTC

Fix proposed to https://review.opendev.org/c/openstack/tooz/+/778878
the gate is broken there, as it seems.

Comment 14 Matthias Runge 2021-06-10 15:11:13 UTC

There is also https://review.opendev.org/c/openstack/tooz/+/795791 to merge for this upstream and then to backport it.

Comment 15 Matthias Runge 2021-07-23 06:42:17 UTC

This is still in upstream review.

Comment 22 Emma Foley 2022-06-22 14:03:21 UTC

There is a backport to Ussuri in progress upstream: https://review.opendev.org/c/openstack/tooz/+/847152

Once the Ussuri backport is merged, the train one can be proposed.

Comment 23 Emma Foley 2022-07-19 16:24:38 UTC

The patch has been backported to train: https://review.opendev.org/c/openstack/tooz/+/848654

Comment 25 Martin Magr 2022-09-07 16:44:16 UTC

This issue has the same cause and fix as BZ1924373

Comment 29 Leonid Natapov 2022-11-06 04:24:22 UTC

No errors or traces appear in ceilometer logs.

Comment 35 errata-xmlrpc 2022-12-07 20:24:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795