Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1791243

Summary: OSP16 | legacy Telemetry | Aprox 24 hours after deploying Openstack with legacy telemetry,gnocchi stops processing metrics.
Product: Red Hat OpenStack Reporter: Leonid Natapov <lnatapov>
Component: openstack-gnocchiAssignee: Martin Magr <mmagr>
Status: CLOSED NOTABUG QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: unspecified    
Version: 16.0 (Train)CC: apevec, jschluet, lhh, pkilambi
Target Milestone: gaKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-15 10:51:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Leonid Natapov 2020-01-15 10:21:15 UTC 
Description of problem:
OSP16 | legacy Telemetry | Aprox 24 hours after deploying Openstack with legacy telemetry,gnocchi stops processing metrics.


Each openstack command related to metric gives the following errors:
--------------------------------------------------------------------

(overcloud) [stack@undercloud-0 ~]$ openstack metric list
Unable to establish connection to http://10.0.0.148:8041/v1/metric?: HTTPConnectionPool(host='10.0.0.148', port=8041): Max retries exceeded with url: /v1/metric (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1e04990780>: Failed to establish a new connection: [Errno 111] Connection refused',))


gnocchi-metricd.log has the following errors:
--------------------------------------------------------------------


2020-01-13 03:59:01,802 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-119
2020-01-13 03:59:01,803 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-120
2020-01-13 03:59:01,804 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-121
2020-01-13 03:59:01,805 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-122
2020-01-13 03:59:01,806 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-123
2020-01-13 03:59:01,807 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-124
2020-01-13 03:59:01,808 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-125
2020-01-13 03:59:01,809 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-126
2020-01-13 03:59:01,810 [31] DEBUG    gnocchi.chef: Processing measures for sack incoming128-127
2020-01-13 03:59:01,811 [31] DEBUG    gnocchi.cli.metricd: 0 metrics processed from 128 sacks
2020-01-13 03:59:01,816 [31] DEBUG    gnocchi.cli.metricd: Full scan of sacks has been done
2020-01-13 03:59:18,058 [31] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 184, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2020-01-13 03:59:18,917 [28] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 184, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2020-01-13 03:59:31,530 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-0
2020-01-13 03:59:31,531 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-1
2020-01-13 03:59:31,532 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-2
2020-01-13 03:59:31,533 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-3
2020-01-13 03:59:31,535 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-4
2020-01-13 03:59:31,536 [28] DEBUG    gnocchi.chef: Processing measures for sack incoming128-5