Bug 2070565 - [RHOSP 16.2] Gnocchi throwing errors when haproxy closes TCP connections to Redis
Summary: [RHOSP 16.2] Gnocchi throwing errors when haproxy closes TCP connections to R...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 16.2 (Train)
Hardware: Unspecified
OS: Linux
high
high
Target Milestone: z4
: 16.2 (Train on RHEL 8.4)
Assignee: Martin Magr
QA Contact: Leonid Natapov
Joanne O'Flynn
URL:
Whiteboard:
Depends On: 1924373
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-31 12:51 UTC by Martin Magr
Modified: 2023-01-03 21:37 UTC (History)
27 users (show)

Fixed In Version: puppet-tripleo-11.7.0-2.20220806005428.47ef865.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of: 1924373
Environment:
Last Closed: 2022-12-07 19:22:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-14418 0 None None None 2022-03-31 12:54:58 UTC
Red Hat Product Errata RHBA-2022:8794 0 None None None 2022-12-07 19:22:54 UTC

Comment 8 Leonid Natapov 2022-05-11 05:53:15 UTC
puppet-tripleo-11.7.0-2.20220405015036.f404729.el8ost.noarch - no errors seen in gnocchi-metricd

Comment 9 Leonid Natapov 2022-05-23 14:35:29 UTC
After checking gnocchi-statsd log file (not gnocchi-metricd.log like described in the bug) I see the same error.
Re-opening this BZ.

version that I checked it is puppet-tripleo-11.7.0-2.20220405015037.el8ost.noarch


2022-05-23 12:52:58,329 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-99
2022-05-23 12:52:58,331 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-104
2022-05-23 12:52:58,333 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-106
2022-05-23 12:52:58,336 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-108
2022-05-23 12:52:58,338 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-109
2022-05-23 12:52:58,340 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-115
2022-05-23 12:52:58,342 [16] DEBUG    gnocchi.chef: Processing measures for sack incoming128-122
2022-05-23 12:52:58,344 [16] DEBUG    gnocchi.cli.metricd: 0 metrics processed from 43 sacks
2022-05-23 12:52:58,344 [16] DEBUG    gnocchi.cli.metricd: Full scan of sacks has been done
2022-05-23 12:52:58,827 [14] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.9/site-packages/gnocchi/incoming/redis.py", line 199, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3605, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3505, in parse_response
    response = self._execute(conn, conn.read_response)
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3479, in _execute
    return command(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
2022-05-23 12:52:58,831 [10] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.9/site-packages/gnocchi/incoming/redis.py", line 199, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3605, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3505, in parse_response
    response = self._execute(conn, conn.read_response)
  File "/usr/lib/python3.9/site-packages/redis/client.py", line 3479, in _execute
    return command(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/lib/python3.9/site-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

Comment 11 Leif Madsen 2022-05-30 14:12:43 UTC
Moving to the backlog and reducing priority as we have no developers available to move this forward currently.

Comment 20 Leonid Natapov 2022-11-01 11:27:21 UTC
I have an OSP16z9 environment for 20 hours now and no traces,errors,fail messages appear in gnocchi-metricd or gnocchi-statsd log files.

haproxy.pp:

$redis_tcp_check_options = $redis_tcp_check_connect_options + $redis_tcp_check_common_options
    haproxy::listen { 'redis':
      bind             => $redis_bind_opts,
      options          => {
        'balance'        => 'first',
        'timeout client' => '90m',
        'timeout server' => '90m',
        'option'         => [ 'tcp-check', 'tcplog', ],
        'tcp-check'      => $redis_tcp_check_options,



Verifying this bug.

Comment 26 errata-xmlrpc 2022-12-07 19:22:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794


Note You need to log in before you can comment on or make changes to this bug.