Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1813771 - telemetry_tempest_plugin.scenario.test_gnocchi.GnocchiGabbiTest.test_live is failed [NEEDINFO]
Summary: telemetry_tempest_plugin.scenario.test_gnocchi.GnocchiGabbiTest.test_live is ...
Keywords:
Status: ON_QA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: gnocchi
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
medium
low
Target Milestone: z1
: 16.2 (Train on RHEL 8.4)
Assignee: Martin Magr
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-16 02:13 UTC by Ryo Hayakawa
Modified: 2021-07-18 09:59 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
lnatapov: needinfo? (mmagr)


Attachments (Terms of Use)

Comment 3 Matthias Runge 2020-08-19 15:31:48 UTC
Is this still an issue? Is this connected to the use of NFS and selinux attributes?

Comment 17 Matthias Runge 2021-02-01 12:12:30 UTC
The error message in Leonids reproducer is:

ft1.1: telemetry_tempest_plugin.scenario.test_gnocchi.GnocchiGabbiTest.test_livetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 61, in test
    run_test(self, test_dir, filename)
  File "/usr/lib/python3.6/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 53, in run_test
    test_class_instance.fail(msg)
  File "/usr/lib/python3.6/site-packages/unittest2/case.py", line 693, in fail
    raise self.failureException(msg)
AssertionError: From test "get myresource measures with refresh" :
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gabbi/suitemaker.py", line 95, in do_test
    return test_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 94, in wrapper
    func(self)
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 148, in test_request
    self._run_test()
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 555, in _run_test
    self._assert_response()
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 193, in _assert_response
    self._test_status(self.test_data['status'], self.response['status'])
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 610, in _test_status
    self.assert_in_or_print_output(observed_status, statii)
  File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 671, in assert_in_or_print_output
    self.fail(msg)
  File "/usr/lib/python3.6/site-packages/unittest2/case.py", line 693, in fail
    raise self.failureException(msg)
AssertionError: '500' not found in ['200'], response:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Comment 18 Leif Madsen 2021-02-01 14:00:13 UTC
I don't see anything in particular that looks related, but since a 500 error in gnocchi was just resolved, thought I'd at least mention it: https://bugzilla.redhat.com/show_bug.cgi?id=1883733

Comment 19 Matthias Runge 2021-02-03 10:36:58 UTC
In Leonids deployment, I see stack traces in /var/log/containers/gnocchi/app.log:
----------------------------------------------------------------------
2021-02-02 23:21:42,919 [26] ERROR    gnocchi.chef: Error processing new measures
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/chef.py", line 124, in refresh_metrics
    in six.iteritems(metrics_and_measures)
  File "/usr/lib/python3.6/site-packages/gnocchi/storage/__init__.py", line 714, in add_measures_to_metrics
    before_truncate_callback=_map_compute_splits_operations,
  File "/usr/lib/python3.6/site-packages/gnocchi/carbonara.py", line 364, in set_values
    return_value = before_truncate_callback(self)
  File "/usr/lib/python3.6/site-packages/gnocchi/storage/__init__.py", line 673, in _map_compute_splits_operations
    tstamp = max(bound_timeserie.first, measures['timestamps'][0])
IndexError: index 0 is out of bounds for axis 0 with size 0
2021-02-02 23:25:59,425 [27] WARNING  py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_805032245a134ae3860b21a7713da1e3_resource, and will be replaced in the string-lookup table.
  % (item.__module__, item.__name__)

2021-02-02 23:25:59,431 [27] WARNING  py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_805032245a134ae3860b21a7713da1e3_history, and will be replaced in the string-lookup table.
  % (item.__module__, item.__name__)

2021-02-03 07:19:31,278 [25] ERROR    pecan.core: Controller 'get' defined does not support content_type 'None'. Supported type(s): dict_keys(['application/json'])
2021-02-03 07:19:34,006 [28] ERROR    pecan.core: Controller 'get_all' defined does not support content_type 'None'. Supported type(s): dict_keys(['application/json'])
2021-02-03 07:19:54,164 [26] WARNING  py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_d724012078f54651ae9324dfeb5904e6_resource, and will be replaced in the string-lookup table.
  % (item.__module__, item.__name__)

2021-02-03 07:19:54,174 [26] WARNING  py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_d724012078f54651ae9324dfeb5904e6_history, and will be replaced in the string-lookup table.
  % (item.__module__, item.__name__)
--------------------------------------------------------

and in gnocchi/gnocchi_metricd.log:
----------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2021-02-03 10:24:26,018 [40] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retryi
ng
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)


These redis issues are all over in all logs (api log as well).

Redis itself did not complain, and it's not gone for all the time.

Comment 20 Matthias Runge 2021-02-03 18:53:43 UTC
The redis issue may be a red herring here, see https://bugzilla.redhat.com/show_bug.cgi?id=1924373


Note You need to log in before you can comment on or make changes to this bug.