Bug 2070565
| Summary: | [RHOSP 16.2] Gnocchi throwing errors when haproxy closes TCP connections to Redis | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Martin Magr <mmagr> |
| Component: | puppet-tripleo | Assignee: | Martin Magr <mmagr> |
| Status: | CLOSED ERRATA | QA Contact: | Leonid Natapov <lnatapov> |
| Severity: | high | Docs Contact: | Joanne O'Flynn <joflynn> |
| Priority: | high | ||
| Version: | 16.2 (Train) | CC: | apevec, astupnik, bdobreli, bperkins, dciabrin, dhill, dhruv, dsedgmen, eulim, fgadkano, jeckersb, jhardee, jjoyce, jschluet, kmehta, lhh, lmadsen, lmiccini, lnatapov, mrunge, mvalsecc, oblaut, sboyron, slinaber, tkajinam, tvignaud, ykulkarn |
| Target Milestone: | z4 | Keywords: | Triaged, ZStream |
| Target Release: | 16.2 (Train on RHEL 8.4) | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | puppet-tripleo-11.7.0-2.20220806005428.47ef865.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1924373 | Environment: | |
| Last Closed: | 2022-12-07 19:22:17 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1924373 | ||
| Bug Blocks: | |||
|
Comment 8
Leonid Natapov
2022-05-11 05:53:15 UTC
After checking gnocchi-statsd log file (not gnocchi-metricd.log like described in the bug) I see the same error.
Re-opening this BZ.
version that I checked it is puppet-tripleo-11.7.0-2.20220405015037.el8ost.noarch
2022-05-23 12:52:58,329 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-99
2022-05-23 12:52:58,331 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-104
2022-05-23 12:52:58,333 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-106
2022-05-23 12:52:58,336 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-108
2022-05-23 12:52:58,338 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-109
2022-05-23 12:52:58,340 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-115
2022-05-23 12:52:58,342 [16] DEBUG gnocchi.chef: Processing measures for sack incoming128-122
2022-05-23 12:52:58,344 [16] DEBUG gnocchi.cli.metricd: 0 metrics processed from 43 sacks
2022-05-23 12:52:58,344 [16] DEBUG gnocchi.cli.metricd: Full scan of sacks has been done
2022-05-23 12:52:58,827 [14] ERROR gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
for sack in self.incoming.iter_on_sacks_to_process():
File "/usr/lib/python3.9/site-packages/gnocchi/incoming/redis.py", line 199, in iter_on_sacks_to_process
for message in p.listen():
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3605, in listen
response = self.handle_message(self.parse_response(block=True))
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3505, in parse_response
response = self._execute(conn, conn.read_response)
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3479, in _execute
return command(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
response = self._parser.read_response()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
raw = self._buffer.readline()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
self._read_from_socket()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2022-05-23 12:52:58,831 [10] ERROR gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
for sack in self.incoming.iter_on_sacks_to_process():
File "/usr/lib/python3.9/site-packages/gnocchi/incoming/redis.py", line 199, in iter_on_sacks_to_process
for message in p.listen():
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3605, in listen
response = self.handle_message(self.parse_response(block=True))
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3505, in parse_response
response = self._execute(conn, conn.read_response)
File "/usr/lib/python3.9/site-packages/redis/client.py", line 3479, in _execute
return command(*args, **kwargs)
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
response = self._parser.read_response()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
raw = self._buffer.readline()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
self._read_from_socket()
File "/usr/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
Moving to the backlog and reducing priority as we have no developers available to move this forward currently. I have an OSP16z9 environment for 20 hours now and no traces,errors,fail messages appear in gnocchi-metricd or gnocchi-statsd log files.
haproxy.pp:
$redis_tcp_check_options = $redis_tcp_check_connect_options + $redis_tcp_check_common_options
haproxy::listen { 'redis':
bind => $redis_bind_opts,
options => {
'balance' => 'first',
'timeout client' => '90m',
'timeout server' => '90m',
'option' => [ 'tcp-check', 'tcplog', ],
'tcp-check' => $redis_tcp_check_options,
Verifying this bug.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8794 |