FFU | While FFU from osp16.2 to osp17.1 overcloud update fails on ceilometer_gnocchi_upgrade OSP16.2 - core_puddle: RHOS-16.2-RHEL-8-20230301.n.1 OSP17.1 - core_puddle: RHOS-17.1-RHEL-8-20230314.n.1 Error message from from the overcloud update log: ---------------------------------------- 14 16:59:42 controller-0 ansible-tripleo_container_manage[930823]: [WARNING] ERROR: Can't run container ceilometer_gnocchi_upgrade#012stderr: + sudo -E kolla_set_configs#012sudo: unable to send audit message: Operation not permitted#012INFO: Errors from the ceilometer_upgrade.log file: -------------------------------------------- 2023-03-14 16:54:28.761 15 ERROR ceilometer Traceback (most recent call last): 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module> 2023-03-14 16:54:28.761 15 ERROR ceilometer sys.exit(upgrade()) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/ceilometer/cmd/storage.py", line 49, in upgrade 2023-03-14 16:54:28.761 15 ERROR ceilometer tenacity.Retrying( 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2023-03-14 16:54:28.761 15 ERROR ceilometer do = self.iter(retry_state=retry_state) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2023-03-14 16:54:28.761 15 ERROR ceilometer return fut.result() 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.__get_result() 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2023-03-14 16:54:28.761 15 ERROR ceilometer raise self._exception 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2023-03-14 16:54:28.761 15 ERROR ceilometer result = fn(*args, **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/ceilometer/gnocchi_client.py", line 259, in upgrade_resource_types 2023-03-14 16:54:28.761 15 ERROR ceilometer gnocchi_version = version.LooseVersion(gnocchi.build.get()) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/build.py", line 19, in get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self._get("").json().get("build", "unknown") 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/base.py", line 37, in _get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.client.api.get(*args, **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.request(url, 'GET', **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/client.py", line 52, in request 2023-03-14 16:54:28.761 15 ERROR ceilometer raise exceptions.from_response(resp, method) 2023-03-14 16:54:28.761 15 ERROR ceilometer gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2023-03-14 16:54:28.761 15 ERROR ceilometer <html><head> 2023-03-14 16:54:28.761 15 ERROR ceilometer <title>504 Gateway Timeout</title> 2023-03-14 16:54:28.761 15 ERROR ceilometer </head><body> 2023-03-14 16:54:28.761 15 ERROR ceilometer <h1>Gateway Timeout</h1> 2023-03-14 16:54:28.761 15 ERROR ceilometer <p>The gateway did not receive a timely response 2023-03-14 16:54:28.761 15 ERROR ceilometer from the upstream server or application.</p> 2023-03-14 16:54:28.761 15 ERROR ceilometer </body></html> 2023-03-14 16:54:28.761 15 ERROR ceilometer (HTTP 504) 2023-03-14 16:54:28.761 15 ERROR ceilometer 2023-03-14 16:54:59.847 17 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3.9/site-packages/ceilometer/cmd/storage.py:42 2023-03-14 16:55:00.237 17 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> (HTTP 503)
Gnocchi api reports "Unable to initialize coordination driver" which is redis ~~~ 2023-03-14 14:10:36,235 [16] ERROR gnocchi.utils: Unable to initialize coordination driver Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 57, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 484, in _start self._server_info = self._client.info() File "/usr/lib/python3.6/site-packages/redis/client.py", line 1036, in info return self.execute_command('INFO') File "/usr/lib/python3.6/site-packages/redis/client.py", line 836, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/usr/lib/python3.6/site-packages/redis/connection.py", line 1071, in get_connection connection.connect() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 547, in connect self.on_connect() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 617, in on_connect if nativestr(self.read_response()) != 'OK': File "/usr/lib/python3.6/site-packages/redis/connection.py", line 699, in read_response response = self._parser.read_response() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 309, in read_response response = self._buffer.readline() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 241, in readline self._read_from_socket() File "/usr/lib/python3.6/site-packages/redis/connection.py", line 186, in _read_from_socket raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR) redis.exceptions.ConnectionError: Connection closed by server. ~~~ Redis status from pacemaker ~~~ * Container bundle set: redis-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-redis:pcmklatest]: * redis-bundle-0 (ocf::heartbeat:redis): Stopped (disabled) * redis-bundle-1 (ocf::heartbeat:redis): Stopped (disabled) * redis-bundle-2 (ocf::heartbeat:redis): Stopped (disabled) ~~~ Probably because redis was stopped, ceilometer upgrade command didn't execute successfully.
So,redis is deprecated and disabled by default in OSP17.1. That's why redis stopped. In OSP16.2 we didn't include any templates for enabling redis. I would expect things that worked in OSP16.2 keep working in OSP17.1
We actually passed enable-legacy-telemetry.yaml template in deploy command This template includes reference to redis-pacemaker-puppet.yaml that must be included in deploy command in oder to enable Redis. So, basically we did passed all necessary templates I checked the overcloud_deploy.sh and it includes --environment-file /usr/share/openstack-tripleo-heat-templates/environments/enable-legacy-telemetry.yaml
green field deployments work
Redis is preserved in case of telemetry after FFU from 16.2 to 17.1 and in MultiRhel FFU from 17.1 RHEL8 to 17.1 RHEL9
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577