Is this still an issue? Is this connected to the use of NFS and selinux attributes?
The error message in Leonids reproducer is: ft1.1: telemetry_tempest_plugin.scenario.test_gnocchi.GnocchiGabbiTest.test_livetesttools.testresult.real._StringException: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 61, in test run_test(self, test_dir, filename) File "/usr/lib/python3.6/site-packages/telemetry_tempest_plugin/scenario/utils.py", line 53, in run_test test_class_instance.fail(msg) File "/usr/lib/python3.6/site-packages/unittest2/case.py", line 693, in fail raise self.failureException(msg) AssertionError: From test "get myresource measures with refresh" : testtools.testresult.real._StringException: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/gabbi/suitemaker.py", line 95, in do_test return test_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 94, in wrapper func(self) File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 148, in test_request self._run_test() File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 555, in _run_test self._assert_response() File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 193, in _assert_response self._test_status(self.test_data['status'], self.response['status']) File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 610, in _test_status self.assert_in_or_print_output(observed_status, statii) File "/usr/lib/python3.6/site-packages/gabbi/case.py", line 671, in assert_in_or_print_output self.fail(msg) File "/usr/lib/python3.6/site-packages/unittest2/case.py", line 693, in fail raise self.failureException(msg) AssertionError: '500' not found in ['200'], response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html>
I don't see anything in particular that looks related, but since a 500 error in gnocchi was just resolved, thought I'd at least mention it: https://bugzilla.redhat.com/show_bug.cgi?id=1883733
In Leonids deployment, I see stack traces in /var/log/containers/gnocchi/app.log: ---------------------------------------------------------------------- 2021-02-02 23:21:42,919 [26] ERROR gnocchi.chef: Error processing new measures Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/gnocchi/chef.py", line 124, in refresh_metrics in six.iteritems(metrics_and_measures) File "/usr/lib/python3.6/site-packages/gnocchi/storage/__init__.py", line 714, in add_measures_to_metrics before_truncate_callback=_map_compute_splits_operations, File "/usr/lib/python3.6/site-packages/gnocchi/carbonara.py", line 364, in set_values return_value = before_truncate_callback(self) File "/usr/lib/python3.6/site-packages/gnocchi/storage/__init__.py", line 673, in _map_compute_splits_operations tstamp = max(bound_timeserie.first, measures['timestamps'][0]) IndexError: index 0 is out of bounds for axis 0 with size 0 2021-02-02 23:25:59,425 [27] WARNING py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_805032245a134ae3860b21a7713da1e3_resource, and will be replaced in the string-lookup table. % (item.__module__, item.__name__) 2021-02-02 23:25:59,431 [27] WARNING py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_805032245a134ae3860b21a7713da1e3_history, and will be replaced in the string-lookup table. % (item.__module__, item.__name__) 2021-02-03 07:19:31,278 [25] ERROR pecan.core: Controller 'get' defined does not support content_type 'None'. Supported type(s): dict_keys(['application/json']) 2021-02-03 07:19:34,006 [28] ERROR pecan.core: Controller 'get_all' defined does not support content_type 'None'. Supported type(s): dict_keys(['application/json']) 2021-02-03 07:19:54,164 [26] WARNING py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_d724012078f54651ae9324dfeb5904e6_resource, and will be replaced in the string-lookup table. % (item.__module__, item.__name__) 2021-02-03 07:19:54,174 [26] WARNING py.warnings: /usr/lib64/python3.6/site-packages/sqlalchemy/ext/declarative/clsregistry.py:129: SAWarning: This declarative base already contains a class with the same class name and module name as gnocchi.indexer.sqlalchemy.rt_d724012078f54651ae9324dfeb5904e6_history, and will be replaced in the string-lookup table. % (item.__module__, item.__name__) -------------------------------------------------------- and in gnocchi/gnocchi_metricd.log: ---------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process for sack in self.incoming.iter_on_sacks_to_process(): File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process for message in p.listen(): File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen response = self.handle_message(self.parse_response(block=True)) File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response return self._execute(connection, connection.read_response) File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute return command(*args) File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response raise e File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response response = self._parser.read_response() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response response = self._buffer.readline() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline self._read_from_socket() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket (e.args,)) redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',) 2021-02-03 10:24:26,018 [40] ERROR gnocchi.cli.metricd: Error while listening for new measures notification, retryi ng Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket raise socket.error(SERVER_CLOSED_CONNECTION_ERROR) OSError: Connection closed by server. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process for sack in self.incoming.iter_on_sacks_to_process(): File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process for message in p.listen(): File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen response = self.handle_message(self.parse_response(block=True)) File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response return self._execute(connection, connection.read_response) File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute return command(*args) File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response raise e File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response response = self._parser.read_response() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response response = self._buffer.readline() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline self._read_from_socket() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket (e.args,)) redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',) These redis issues are all over in all logs (api log as well). Redis itself did not complain, and it's not gone for all the time.
The redis issue may be a red herring here, see https://bugzilla.redhat.com/show_bug.cgi?id=1924373
test passes on OSP 16.2