Bug 2178509
| Summary: | FFU | While FFU from osp16.2 to osp17.1 overcloud update fails on ceilometer_gnocchi_upgrade | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Leonid Natapov <lnatapov> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Juan Badia Payno <jbadiapa> |
| Status: | CLOSED ERRATA | QA Contact: | Leonid Natapov <lnatapov> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 17.1 (Wallaby) | CC: | apevec, jbadiapa, jpretori, jstransk, kthakre, lmadsen, lmiccini, mburns, mrunge, pgrist, ykulkarn |
| Target Milestone: | ga | Keywords: | Triaged |
| Target Release: | 17.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-14.3.1-1.20230519151007.el9ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-08-16 01:14:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2016660, 2158159 | ||
Gnocchi api reports "Unable to initialize coordination driver" which is redis
~~~
2023-03-14 14:10:36,235 [16] ERROR gnocchi.utils: Unable to initialize coordination driver
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 57, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/tooz/drivers/redis.py", line 484, in _start
self._server_info = self._client.info()
File "/usr/lib/python3.6/site-packages/redis/client.py", line 1036, in info
return self.execute_command('INFO')
File "/usr/lib/python3.6/site-packages/redis/client.py", line 836, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 1071, in get_connection
connection.connect()
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 547, in connect
self.on_connect()
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 617, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 699, in read_response
response = self._parser.read_response()
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 309, in read_response
response = self._buffer.readline()
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 241, in readline
self._read_from_socket()
File "/usr/lib/python3.6/site-packages/redis/connection.py", line 186, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
~~~
Redis status from pacemaker
~~~
* Container bundle set: redis-bundle [undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-redis:pcmklatest]:
* redis-bundle-0 (ocf::heartbeat:redis): Stopped (disabled)
* redis-bundle-1 (ocf::heartbeat:redis): Stopped (disabled)
* redis-bundle-2 (ocf::heartbeat:redis): Stopped (disabled)
~~~
Probably because redis was stopped, ceilometer upgrade command didn't execute successfully.
So,redis is deprecated and disabled by default in OSP17.1. That's why redis stopped. In OSP16.2 we didn't include any templates for enabling redis. I would expect things that worked in OSP16.2 keep working in OSP17.1 We actually passed enable-legacy-telemetry.yaml template in deploy command This template includes reference to redis-pacemaker-puppet.yaml that must be included in deploy command in oder to enable Redis. So, basically we did passed all necessary templates I checked the overcloud_deploy.sh and it includes --environment-file /usr/share/openstack-tripleo-heat-templates/environments/enable-legacy-telemetry.yaml green field deployments work Redis is preserved in case of telemetry after FFU from 16.2 to 17.1 and in MultiRhel FFU from 17.1 RHEL8 to 17.1 RHEL9 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577 |
FFU | While FFU from osp16.2 to osp17.1 overcloud update fails on ceilometer_gnocchi_upgrade OSP16.2 - core_puddle: RHOS-16.2-RHEL-8-20230301.n.1 OSP17.1 - core_puddle: RHOS-17.1-RHEL-8-20230314.n.1 Error message from from the overcloud update log: ---------------------------------------- 14 16:59:42 controller-0 ansible-tripleo_container_manage[930823]: [WARNING] ERROR: Can't run container ceilometer_gnocchi_upgrade#012stderr: + sudo -E kolla_set_configs#012sudo: unable to send audit message: Operation not permitted#012INFO: Errors from the ceilometer_upgrade.log file: -------------------------------------------- 2023-03-14 16:54:28.761 15 ERROR ceilometer Traceback (most recent call last): 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module> 2023-03-14 16:54:28.761 15 ERROR ceilometer sys.exit(upgrade()) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/ceilometer/cmd/storage.py", line 49, in upgrade 2023-03-14 16:54:28.761 15 ERROR ceilometer tenacity.Retrying( 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__ 2023-03-14 16:54:28.761 15 ERROR ceilometer do = self.iter(retry_state=retry_state) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter 2023-03-14 16:54:28.761 15 ERROR ceilometer return fut.result() 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.__get_result() 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2023-03-14 16:54:28.761 15 ERROR ceilometer raise self._exception 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__ 2023-03-14 16:54:28.761 15 ERROR ceilometer result = fn(*args, **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/ceilometer/gnocchi_client.py", line 259, in upgrade_resource_types 2023-03-14 16:54:28.761 15 ERROR ceilometer gnocchi_version = version.LooseVersion(gnocchi.build.get()) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/build.py", line 19, in get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self._get("").json().get("build", "unknown") 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/v1/base.py", line 37, in _get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.client.api.get(*args, **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/keystoneauth1/adapter.py", line 395, in get 2023-03-14 16:54:28.761 15 ERROR ceilometer return self.request(url, 'GET', **kwargs) 2023-03-14 16:54:28.761 15 ERROR ceilometer File "/usr/lib/python3.9/site-packages/gnocchiclient/client.py", line 52, in request 2023-03-14 16:54:28.761 15 ERROR ceilometer raise exceptions.from_response(resp, method) 2023-03-14 16:54:28.761 15 ERROR ceilometer gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2023-03-14 16:54:28.761 15 ERROR ceilometer <html><head> 2023-03-14 16:54:28.761 15 ERROR ceilometer <title>504 Gateway Timeout</title> 2023-03-14 16:54:28.761 15 ERROR ceilometer </head><body> 2023-03-14 16:54:28.761 15 ERROR ceilometer <h1>Gateway Timeout</h1> 2023-03-14 16:54:28.761 15 ERROR ceilometer <p>The gateway did not receive a timely response 2023-03-14 16:54:28.761 15 ERROR ceilometer from the upstream server or application.</p> 2023-03-14 16:54:28.761 15 ERROR ceilometer </body></html> 2023-03-14 16:54:28.761 15 ERROR ceilometer (HTTP 504) 2023-03-14 16:54:28.761 15 ERROR ceilometer 2023-03-14 16:54:59.847 17 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3.9/site-packages/ceilometer/cmd/storage.py:42 2023-03-14 16:55:00.237 17 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> (HTTP 503)