Bug 1957979
| Summary: | Getting Gateway Timeout (HTTP 504) after restart the RHOSP 16.1 All-In-One server with default deployment | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexon Oliveira <alolivei> |
| Component: | openstack-tripleo-heat-templates | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | Joe H. Rahme <jhakimra> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 16.1 (Train) | CC: | bdobreli, dabarzil, dciabrin, lmiccini, mbayer, mburns, rhos-maint |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-25 12:39:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi, Judging from the output of pcs status there are at least a couple of things that are out of shape: 1. lack of "/usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml" so you don't have all the pacemaker resources defined properly 2. missing the control plane VIP. Regarding 1. : this is automatically included when you deploy a overcloud via the plan-environment.yaml file and I guess the standalone hasn't been adjusted to reflect this. Regarding 2. : this should be passed as an option to the deploy command Can you try adjusting your deployment command to look something like this: sudo openstack tripleo deploy --templates --local-ip=IP/NETMASK --control-virtual-ip=192.168.24.3 \ -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ .... Please let us know if that helps. Hi Luca, Thanks for the heads up. Just FYI, the deployment commands I ran was accordingly to the chapter 5 of the all-in-one 16.1 deployment official doc [1], like this: === [stack@openstack16 ~]$ cat deploy.sh #!/bin/bash sudo openstack tripleo deploy --templates --local-ip=$IP/$NETMASK \ -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \ -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ -e $HOME/containers-prepare-parameters.yaml \ -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone === [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/standalone_deployment_guide/deploying-the-all-in-one-openstack-installation I've changed it as you asked me to do, like the following: === [stack@openstack16 ~]$ cat deploy.sh #!/bin/bash sudo openstack tripleo deploy --templates --control-virtual-ip=$IP --local-ip=$IP/$NETMASK \ -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ -e $HOME/containers-prepare-parameters.yaml \ -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone === I ran a fresh newly deploy again with the adjusted deploy.sh file with the commands you've recommended and it ran with success, as follows: ===output ommited==== PLAY RECAP ****************************************************************************************************************************************** openstack16 : ok=341 changed=146 unreachable=0 failed=0 skipped=125 rescued=0 ignored=0 undercloud : ok=84 changed=30 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0 Monday 10 May 2021 16:36:22 +0000 (0:00:00.063) 0:28:57.346 ************ =============================================================================== Wait for container-puppet tasks (generate config) to finish -------------------------------------------------------------------------------- 350.63s Wait for containers to start for step 4 using paunch --------------------------------------------------------------------------------------- 254.35s Wait for containers to start for step 2 using paunch --------------------------------------------------------------------------------------- 180.20s Wait for containers to start for step 3 using paunch --------------------------------------------------------------------------------------- 120.67s Wait for containers to start for step 5 using paunch --------------------------------------------------------------------------------------- 112.23s Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 58.09s Wait for container-puppet tasks (bootstrap tasks) for step 4 to finish ---------------------------------------------------------------------- 54.18s Manage Cinder Volume Type ------------------------------------------------------------------------------------------------------------------- 30.98s Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 29.54s Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 27.50s Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 24.86s Run puppet on the host to apply IPtables rules ---------------------------------------------------------------------------------------------- 20.00s Wait for container-puppet tasks (bootstrap tasks) for step 3 to finish ---------------------------------------------------------------------- 17.80s Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 16.76s Pre-fetch all the containers ----------------------------------------------------------------------------------------------------------------- 9.15s tripleo-bootstrap : Deploy required packages to bootstrap TripleO ---------------------------------------------------------------------------- 7.94s tripleo-keystone-resources : Check Keystone service status ----------------------------------------------------------------------------------- 7.42s tripleo-keystone-resources : Check Keystone user assignment to roles status ------------------------------------------------------------------ 7.33s tripleo-keystone-resources : Check Keystone internal endpoint status ------------------------------------------------------------------------- 6.97s tripleo-keystone-resources : Check Keystone admin endpoint status ---------------------------------------------------------------------------- 6.91s Not cleaning working directory /home/stack/tripleo-heat-installer-templates Not cleaning ansible directory /home/stack/standalone-ansible-swk7qpid Install artifact is located at /home/stack/standalone-install-20210510163622.tar.bzip2 ######################################################## Deployment successful! ######################################################## ########################################################## Useful files: The clouds.yaml file is at ~/.config/openstack/clouds.yaml Use "export OS_CLOUD=standalone" before running the openstack command. ########################################################## Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_standalone === I managed to run any openstack command with no problem: === [stack@openstack16 ~]$ openstack service list +----------------------------------+-----------+--------------+ | ID | Name | Type | +----------------------------------+-----------+--------------+ | 2f34a4a42a5848c8b46cc5ab477481c8 | nova | compute | | 541217a20275434ba6bf24e5eda7cc1d | glance | image | | 6f598bf1c7174fb4afe96188d34d46ee | swift | object-store | | 797c9cafca4e495482e935388fa9dada | cinderv3 | volumev3 | | 7be5974cd6864dfeaac622391302f975 | keystone | identity | | 993209dadd1c4385970345d8168b56ec | neutron | network | | c324eaa0ef40471b889acfe77740884f | cinderv2 | volumev2 | | f8807878538649e381796ae7126adba6 | placement | placement | +----------------------------------+-----------+--------------+ [stack@openstack16 ~]$ sudo pcs cluster status Cluster Status: Cluster Summary: * Stack: corosync * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum * Last updated: Mon May 10 18:21:22 2021 * Last change: Mon May 10 16:35:14 2021 by root via cibadmin on openstack16 * 5 nodes configured * 15 resource instances configured Node List: * Online: [ openstack16 ] * GuestOnline: [ galera-bundle-0@openstack16 ovn-dbs-bundle-0@openstack16 rabbitmq-bundle-0@openstack16 redis-bundle-0@openstack16 ] PCSD Status: openstack16: Online [stack@openstack16 ~]$ uptime 18:21:58 up 4:54, 1 user, load average: 1,70, 1,50, 1,49 === Then I did the test of reboot the all-in-one server and try to run any openstack command again, but as you can see below apparently only the pcs resource for galera-bundle has been solved, but not the fact I can't run any openstack command after reboot: === [stack@openstack16 ~]$ uptime 18:29:44 up 4 min, 1 user, load average: 12,83, 7,58, 3,18 [stack@openstack16 ~]$ sudo pcs cluster status Cluster Status: Cluster Summary: * Stack: corosync * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum * Last updated: Mon May 10 18:29:54 2021 * Last change: Mon May 10 16:35:14 2021 by root via cibadmin on openstack16 * 5 nodes configured * 15 resource instances configured Node List: * Online: [ openstack16 ] PCSD Status: openstack16: Online [stack@openstack16 ~]$ openstack service list Gateway Timeout (HTTP 504) [stack@openstack16 ~]$ openstack --debug service list START with options: --debug service list options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='http://192.168.3.2:5000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='default', project_domain_name='', project_id='08f36e84994f4e098abd115466d79b9c', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'} cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} compute API version 2.1, cmd group openstack.compute.v2 identity API version 3, cmd group openstack.identity.v3 image API version 2, cmd group openstack.image.v2 network API version 2, cmd group openstack.network.v2 object_store API version 1, cmd group openstack.object_store.v1 volume API version 3, cmd group openstack.volume.v3 messaging API version 2, cmd group openstack.messaging.v2 database API version 1, cmd group openstack.database.v1 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 data_processing API version 1.1, cmd group openstack.data_processing.v1 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 neutronclient API version 2, cmd group openstack.neutronclient.v2 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 container_infra API version 1, cmd group openstack.container_infra.v1 baremetal API version 1.58, cmd group openstack.baremetal.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 orchestration API version 1, cmd group openstack.orchestration.v1 dns API version 2, cmd group openstack.dns.v2 key_manager API version 1, cmd group openstack.key_manager.v1 event API version 2, cmd group openstack.event.v2 metric API version 1, cmd group openstack.metric.v1 alarming API version 2, cmd group openstack.alarming.v2 Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} command: service list -> openstackclient.identity.v3.service.ListService (auth=True) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} Using auth plugin: password Using parameters {'auth_url': 'http://192.168.3.2:5000', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin', 'project_domain_id': 'default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} Get auth_ref REQ: curl -g -i -X GET http://192.168.3.2:5000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.4 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8" Starting new HTTP connection (1): 192.168.3.2:5000 http://192.168.3.2:5000 "GET / HTTP/1.1" 504 247 RESP: [504] Connection: Keep-Alive Content-Length: 247 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 10 May 2021 18:32:35 GMT Keep-Alive: timeout=15, max=100 Server: Apache RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged. Request returned failure status: 504 Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin authenticated=False) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery disc = Discover(session, url, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data resp = session.get(url, headers=headers, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command return super(OpenStackShell, self).prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref self._plugin = self._do_create_plugin(session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin 'auth_url is correct. %s' % e) keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) clean_up ListService: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin authenticated=False) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery disc = Discover(session, url, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data resp = session.get(url, headers=headers, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command return super(OpenStackShell, self).prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref self._plugin = self._do_create_plugin(session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin 'auth_url is correct. %s' % e) keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) END return value: 1 [stack@openstack16 ~]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 89f275708376 registry.redhat.io/rhosp-rhel8/openstack-cinder-volume:16.1 /bin/bash /usr/lo... 2 minutes ago Up 2 minutes ago openstack-cinder-volume-podman-0 5ba1b5741373 registry.redhat.io/rhosp-rhel8/openstack-ovn-northd:16.1 /bin/bash /usr/lo... 2 minutes ago Up 2 minutes ago ovn-dbs-bundle-podman-0 e5b4340380f2 registry.redhat.io/rhosp-rhel8/openstack-redis:16.1 /bin/bash /usr/lo... 3 minutes ago Up 3 minutes ago redis-bundle-podman-0 1b94c194a443 registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.1 /bin/bash /usr/lo... 4 minutes ago Up 3 minutes ago rabbitmq-bundle-podman-0 21de845dbd17 registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.1 kolla_start 2 hours ago Up 3 minutes ago nova_compute c3449d54f8f1 registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1 kolla_start 2 hours ago Up 4 minutes ago nova_api_cron a84ac13f19bb registry.redhat.io/rhosp-rhel8/openstack-swift-proxy-server:16.1 kolla_start 2 hours ago Up 4 minutes ago swift_proxy 6a493cab06f7 registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1 kolla_start 2 hours ago Up 4 minutes ago nova_metadata 740d8d5b3960 registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1 kolla_start 2 hours ago Up 4 minutes ago nova_api 92efa265b0ee registry.redhat.io/rhosp-rhel8/openstack-glance-api:16.1 kolla_start 2 hours ago Up 4 minutes ago glance_api e526733ed6de registry.redhat.io/rhosp-rhel8/openstack-placement-api:16.1 kolla_start 2 hours ago Up 4 minutes ago placement_api 099ad2995384 registry.redhat.io/rhosp-rhel8/openstack-neutron-metadata-agent-ovn:16.1 kolla_start 2 hours ago Up 7 minutes ago ovn_metadata_agent 86cfafece23d registry.redhat.io/rhosp-rhel8/openstack-ovn-controller:16.1 kolla_start 2 hours ago Up 4 minutes ago ovn_controller 473a9e76e751 registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1 kolla_start 2 hours ago Up 4 minutes ago swift_rsync 7e6c057c23bb registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1 kolla_start 2 hours ago Up 3 minutes ago swift_object_updater bf9b6aa31e60 registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1 kolla_start 2 hours ago Up 4 minutes ago swift_object_server 542fa4615d5e registry.redhat.io/rhosp-rhel8/openstack-swift-proxy-server:16.1 kolla_start 2 hours ago Up 3 minutes ago swift_object_expirer 4b51eed525e3 registry.redhat.io/rhosp-rhel8/openstack-swift-container:16.1 kolla_start 2 hours ago Up 4 minutes ago swift_container_updater 77bced83e17a registry.redhat.io/rhosp-rhel8/openstack-swift-container:16.1 kolla_start 2 hours ago Up 3 minutes ago swift_container_server aa75a0ec8689 registry.redhat.io/rhosp-rhel8/openstack-swift-account:16.1 kolla_start 2 hours ago Up 3 minutes ago swift_account_server dd0b34104073 registry.redhat.io/rhosp-rhel8/openstack-swift-account:16.1 kolla_start 2 hours ago Up 4 minutes ago swift_account_reaper 68245e7185eb registry.redhat.io/rhosp-rhel8/openstack-nova-novncproxy:16.1 kolla_start 2 hours ago Up 4 minutes ago nova_vnc_proxy 629347d0a808 registry.redhat.io/rhosp-rhel8/openstack-nova-scheduler:16.1 kolla_start 2 hours ago Up 42 seconds ago nova_scheduler 8f12ca348c92 registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.1 kolla_start 2 hours ago Up 4 minutes ago nova_migration_target d76cb47095e2 registry.redhat.io/rhosp-rhel8/openstack-nova-conductor:16.1 kolla_start 2 hours ago Up 35 seconds ago nova_conductor cc75b9d18720 registry.redhat.io/rhosp-rhel8/openstack-neutron-server-ovn:16.1 kolla_start 2 hours ago Up 4 minutes ago neutron_api 07b27271752f registry.redhat.io/rhosp-rhel8/openstack-cron:16.1 kolla_start 2 hours ago Up 4 minutes ago logrotate_crond eeb687f1ee3e registry.redhat.io/rhosp-rhel8/openstack-cinder-scheduler:16.1 kolla_start 2 hours ago Up 4 minutes ago cinder_scheduler 445a8e8666f9 registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.1 kolla_start 2 hours ago Up 4 minutes ago cinder_api_cron 502049a4d16d registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.1 kolla_start 2 hours ago Up 7 minutes ago cinder_api 2cfe9182bef3 registry.redhat.io/rhosp-rhel8/openstack-keystone:16.1 kolla_start 3 hours ago Up 4 minutes ago keystone f7a2f0a07a9e registry.redhat.io/rhosp-rhel8/openstack-iscsid:16.1 kolla_start 3 hours ago Up 4 minutes ago iscsid 209ff53e823d registry.redhat.io/rhosp-rhel8/openstack-nova-libvirt:16.1 kolla_start 3 hours ago Up 3 minutes ago nova_libvirt 7dbc2dc463d2 registry.redhat.io/rhosp-rhel8/openstack-nova-libvirt:16.1 kolla_start 4 hours ago Up 4 minutes ago nova_virtlogd d9b6ed6f87ac registry.redhat.io/rhosp-rhel8/openstack-horizon:16.1 kolla_start 4 hours ago Up 4 minutes ago horizon 6bfbc1b9c5e9 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.1 kolla_start 4 hours ago Up 4 minutes ago clustercheck fcf7b19a54d3 registry.redhat.io/rhosp-rhel8/openstack-memcached:16.1 kolla_start 4 hours ago Up 4 minutes ago memcached === I still have the same problem. Any other suggestion? I think one thing you definitely want to fix is to have --control-virtual-ip and --local-ip using different IPs, for example 192.168.24.3 and 192.168.24.2 respectively: sudo openstack tripleo deploy --templates --control-virtual-ip=$IP --local-ip=$IP/$NETMASK \ -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ -e $HOME/containers-prepare-parameters.yaml \ -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone if that fails maybe I can have a look at your env to figure this out? (In reply to Luca Miccini from comment #3) > I think one thing you definitely want to fix is to have --control-virtual-ip > and --local-ip using different IPs, for example 192.168.24.3 and > 192.168.24.2 respectively: > > sudo openstack tripleo deploy --templates --control-virtual-ip=$IP > --local-ip=$IP/$NETMASK \ > -e > /usr/share/openstack-tripleo-heat-templates/environments/standalone/ > standalone-tripleo.yaml \ > -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ > -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ > -e $HOME/containers-prepare-parameters.yaml \ > -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone > > if that fails maybe I can have a look at your env to figure this out? Luca, Different configs, same results. Sure, you can take a look at my configs. What do you want to check? I can attach here the config files you'll need. (In reply to Alexon Oliveira from comment #4) > (In reply to Luca Miccini from comment #3) > > I think one thing you definitely want to fix is to have --control-virtual-ip > > and --local-ip using different IPs, for example 192.168.24.3 and > > 192.168.24.2 respectively: > > > > sudo openstack tripleo deploy --templates --control-virtual-ip=$IP > > --local-ip=$IP/$NETMASK \ > > -e > > /usr/share/openstack-tripleo-heat-templates/environments/standalone/ > > standalone-tripleo.yaml \ > > -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ > > -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ > > -e $HOME/containers-prepare-parameters.yaml \ > > -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone > > > > if that fails maybe I can have a look at your env to figure this out? > > Luca, > > Different configs, same results. Sure, you can take a look at my configs. > What do you want to check? I can attach here the config files you'll need. can you attach or upload a sosreport somewhere? (btw I just realized that the command that I copy/pasted has the same variables for both the VIPs, I hope you used different IPs). Cheers, Luca Full List of Resources:
* Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
* galera (ocf::heartbeat:galera): Stopped
* galera-bundle-podman-0 (ocf::heartbeat:podman): Stopped
* galera-bundle-0 (ocf::pacemaker:remote): Stopped
* Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
* rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started rabbitmq-bundle-0
* rabbitmq-bundle-podman-0 (ocf::heartbeat:podman): Started openstack16
* rabbitmq-bundle-0 (ocf::pacemaker:remote): Started openstack16
* Container bundle: redis-bundle [cluster.common.tag/openstack-redis:pcmklatest]:
* redis (ocf::heartbeat:redis): Master redis-bundle-0
* redis-bundle-podman-0 (ocf::heartbeat:podman): Started openstack16
* redis-bundle-0 (ocf::pacemaker:remote): Started openstack16
* ip-192.168.3.2 (ocf::heartbeat:IPaddr2): Stopped
* Container bundle: haproxy-bundle [cluster.common.tag/openstack-haproxy:pcmklatest]:
* haproxy-bundle-podman-0 (ocf::heartbeat:podman): Stopped
* Container bundle: ovn-dbs-bundle [cluster.common.tag/openstack-ovn-northd:pcmklatest]:
* ovndb_servers (ocf::ovn:ovndb-servers): Master ovn-dbs-bundle-0
* ovn-dbs-bundle-podman-0 (ocf::heartbeat:podman): Started openstack16
* ovn-dbs-bundle-0 (ocf::pacemaker:remote): Started openstack16
* Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
* openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Started openstack16
you haven't followed my advice to use a different Ip address for --control-virtual-ip :)
please try with
sudo openstack tripleo deploy --templates --control-virtual-ip=192.168.3.3 --local-ip=192.168.3.2/24 \
-e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
-e $HOME/containers-prepare-parameters.yaml \
-e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone
(In reply to Luca Miccini from comment #9) > Full List of Resources: > * Container bundle: galera-bundle > [cluster.common.tag/openstack-mariadb:pcmklatest]: > * galera (ocf::heartbeat:galera): Stopped > * galera-bundle-podman-0 (ocf::heartbeat:podman): Stopped > * galera-bundle-0 (ocf::pacemaker:remote): Stopped > * Container bundle: rabbitmq-bundle > [cluster.common.tag/openstack-rabbitmq:pcmklatest]: > * rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started > rabbitmq-bundle-0 > * rabbitmq-bundle-podman-0 (ocf::heartbeat:podman): > Started openstack16 > * rabbitmq-bundle-0 (ocf::pacemaker:remote): Started > openstack16 > * Container bundle: redis-bundle > [cluster.common.tag/openstack-redis:pcmklatest]: > * redis (ocf::heartbeat:redis): Master redis-bundle-0 > * redis-bundle-podman-0 (ocf::heartbeat:podman): Started > openstack16 > * redis-bundle-0 (ocf::pacemaker:remote): Started openstack16 > * ip-192.168.3.2 (ocf::heartbeat:IPaddr2): Stopped > * Container bundle: haproxy-bundle > [cluster.common.tag/openstack-haproxy:pcmklatest]: > * haproxy-bundle-podman-0 (ocf::heartbeat:podman): Stopped > * Container bundle: ovn-dbs-bundle > [cluster.common.tag/openstack-ovn-northd:pcmklatest]: > * ovndb_servers (ocf::ovn:ovndb-servers): Master > ovn-dbs-bundle-0 > * ovn-dbs-bundle-podman-0 (ocf::heartbeat:podman): Started > openstack16 > * ovn-dbs-bundle-0 (ocf::pacemaker:remote): Started > openstack16 > * Container bundle: openstack-cinder-volume > [cluster.common.tag/openstack-cinder-volume:pcmklatest]: > * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): > Started openstack16 > > > you haven't followed my advice to use a different Ip address for > --control-virtual-ip :) > > please try with > > sudo openstack tripleo deploy --templates --control-virtual-ip=192.168.3.3 > --local-ip=192.168.3.2/24 \ > -e > /usr/share/openstack-tripleo-heat-templates/environments/standalone/ > standalone-tripleo.yaml \ > -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ > -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \ > -e $HOME/containers-prepare-parameters.yaml \ > -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone I've done it. Didn't work either. Check the new logs I provided to you in the same place, please. Also, I did a fresh install with these recommendations. It didn't work. |
Description of problem: On a fresh RHOSP 16.1 All-In-One default deployment (Tech Preview), every time the all-in-one server is rebooted, is not possible to authenticate with Keystone or run any openstack command due a "Gateway Timeout (HTTP 504)" error message: $ openstack service list Gateway Timeout (HTTP 504) It seems like paunch is not being able to start some containers properly and pacemaker is failing to start galera-bundle resource either. Also, Horizon and Keystone do not seem to work at all. The only way to make it work properly again is to run the deploy_steps_playbook.yaml or the update_steps_playbook.yaml, but after run them once and reboot again, it does not work anymore. Version-Release number of selected component (if applicable): 16.1 How reproducible: Always Steps to Reproduce: 1. Deploy the all-in-one solution with default options as per the documentation 2. Reboot the all-in-one server 3. Try to authenticate via CLI and run any openstack command. Horizon is not available either. Actual results: Fails (Services failing after reboot) Expected results: Succeeds (Services needs to work as usual after reboot) Additional info: Environment information: === [root@openstack16 ~]# rpm -qa | grep tripleo openstack-tripleo-puppet-elements-11.2.2-1.20201114042506.f061f90.el8ost.noarch python3-tripleoclient-heat-installer-12.3.2-1.20201114043247.1.el8ost.noarch puppet-tripleo-11.5.0-1.20201114030109.el8ost.noarch tripleo-ansible-0.5.1-1.20201114030848.2.el8ost.noarch openstack-tripleo-common-containers-11.4.1-1.20210104173607.el8ost.noarch openstack-tripleo-common-11.4.1-1.20210104173607.el8ost.noarch python3-tripleoclient-12.3.2-1.20201114043247.1.el8ost.noarch ansible-tripleo-ipsec-9.3.0-1.20201113193132.0c8693c.el8ost.noarch ansible-role-tripleo-modify-image-1.2.1-1.20201114004656.1dffa21.el8ost.noarch ansible-tripleo-ipa-0.2.1-1.20201114010033.3bb3c53.el8ost.noarch python3-tripleo-common-11.4.1-1.20210104173607.el8ost.noarch openstack-tripleo-validations-11.3.2-1.20201114040746.el8ost.noarch openstack-tripleo-heat-templates-11.3.2-1.20210104205664.el8ost.2.noarch openstack-tripleo-image-elements-10.6.2-1.20201113215051.7dc0fa1.el8ost.noarch [root@openstack16 ~]# rpm -qa | grep openstack | grep -v tripleo openstack-ironic-python-agent-builder-2.2.0-1.20201114043345.69e41ff.el8ost.noarch python3-openstacksdk-0.36.4-1.20201113235937.76d3b29.el8ost.noarch openstack-octavia-common-5.0.3-1.20201114024052.8c32d2e.el8ost.noarch openstack-heat-monolith-13.1.0-1.20201114032442.48b730a.el8ost.noarch openstack-octavia-diskimage-create-5.0.3-1.20201114024052.8c32d2e.el8ost.noarch python3-openstackclient-4.0.1-1.20201114012609.bff556c.el8ost.noarch puppet-openstacklib-15.4.1-1.20201113204514.5fdf43c.el8ost.noarch python-openstackclient-lang-4.0.1-1.20201114012609.bff556c.el8ost.noarch openstack-heat-common-13.1.0-1.20201114032442.48b730a.el8ost.noarch openstack-heat-agents-1.10.1-1.20201113195131.96b819c.el8ost.noarch openstack-selinux-0.8.24-1.20200914163012.26243bf.el8ost.noarch ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.noarch openstack-heat-api-13.1.0-1.20201114032442.48b730a.el8ost.noarch puppet-openstack_extras-15.4.1-1.20201113215732.371931c.el8ost.noarch openstack-heat-engine-13.1.0-1.20201114032442.48b730a.el8ost.noarch [root@openstack16 ~]# hostnamectl Static hostname: openstack16.example.local Icon name: computer-vm Chassis: vm Machine ID: f85aa0aaf6894ab792bd555197a13152 Boot ID: 1023c63e1117489681daf0bd9595f29d Virtualization: kvm Operating System: Red Hat Enterprise Linux 8.2 (Ootpa) CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA Kernel: Linux 4.18.0-193.47.1.el8_2.x86_64 Architecture: x86-64 === This is the status of a all-in-one server after a reboot: === [stack@openstack16 ~]$ source admin-openrc.sh Please enter your OpenStack Password for project admin as user admin: [stack@openstack16 ~]$ openstack service list Gateway Timeout (HTTP 504) [stack@openstack16 ~]$ openstack service list Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) [stack@openstack16 ~]$ openstack --debug service list START with options: --debug service list options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='http://192.168.3.2:5000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='default', project_domain_name='', project_id='65a612ee274b48fea0941df2da0dc549', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'} cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} compute API version 2.1, cmd group openstack.compute.v2 identity API version 3, cmd group openstack.identity.v3 image API version 2, cmd group openstack.image.v2 network API version 2, cmd group openstack.network.v2 object_store API version 1, cmd group openstack.object_store.v1 volume API version 3, cmd group openstack.volume.v3 messaging API version 2, cmd group openstack.messaging.v2 database API version 1, cmd group openstack.database.v1 tripleoclient API version 1, cmd group openstack.tripleoclient.v1 data_processing API version 1.1, cmd group openstack.data_processing.v1 load_balancer API version 2.0, cmd group openstack.load_balancer.v2 neutronclient API version 2, cmd group openstack.neutronclient.v2 workflow_engine API version 2, cmd group openstack.workflow_engine.v2 container_infra API version 1, cmd group openstack.container_infra.v1 baremetal API version 1.58, cmd group openstack.baremetal.v1 baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1 orchestration API version 1, cmd group openstack.orchestration.v1 dns API version 2, cmd group openstack.dns.v2 key_manager API version 1, cmd group openstack.key_manager.v1 event API version 2, cmd group openstack.event.v2 metric API version 1, cmd group openstack.metric.v1 alarming API version 2, cmd group openstack.alarming.v2 Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} command: service list -> openstackclient.identity.v3.service.ListService (auth=True) Auth plugin password selected auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []} Using auth plugin: password Using parameters {'auth_url': 'http://192.168.3.2:5000', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin', 'project_domain_id': 'default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} Get auth_ref REQ: curl -g -i -X GET http://192.168.3.2:5000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.4 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8" Starting new HTTP connection (1): 192.168.3.2:5000 http://192.168.3.2:5000 "GET / HTTP/1.1" 504 247 RESP: [504] Connection: Keep-Alive Content-Length: 247 Content-Type: text/html; charset=iso-8859-1 Date: Thu, 06 May 2021 21:26:45 GMT Keep-Alive: timeout=15, max=100 Server: Apache RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged. Request returned failure status: 504 Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin authenticated=False) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery disc = Discover(session, url, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data resp = session.get(url, headers=headers, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command return super(OpenStackShell, self).prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref self._plugin = self._do_create_plugin(session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin 'auth_url is correct. %s' % e) keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) clean_up ListService: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin authenticated=False) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery disc = Discover(session, url, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data resp = session.get(url, headers=headers, authenticated=authenticated) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get return self.request(url, 'GET', **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand self.prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command return super(OpenStackShell, self).prepare_to_run_command(cmd) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command self.client_manager.auth_ref File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref self._auth_ref = self.auth.get_auth_ref(self.session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref self._plugin = self._do_create_plugin(session) File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin 'auth_url is correct. %s' % e) keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504) END return value: 1 [stack@openstack16 ~]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 66478086daed satellite.example.local:5000/example_org-osp16_1_containers-cinder-volume:16.1 /bin/bash /usr/lo... 3 minutes ago Up 3 minutes ago openstack-cinder-volume-podman-0 5c9f5c9b5f12 satellite.example.local:5000/example_org-osp16_1_containers-rabbitmq:16.1 /bin/bash /usr/lo... 5 minutes ago Up 4 minutes ago rabbitmq-bundle-podman-0 b58a332178ab satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1 kolla_start 25 hours ago Up 5 minutes ago swift_proxy 70b451a661f1 satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 25 hours ago Up 3 minutes ago swift_rsync 0c07391fd692 satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 25 hours ago Up 4 minutes ago swift_object_updater 97c91bd9ad6c satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 25 hours ago Up 4 minutes ago swift_object_server ca8a2a5a13ee satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1 kolla_start 25 hours ago Up 5 minutes ago swift_object_expirer f54832f70f51 satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1 kolla_start 25 hours ago Up 4 minutes ago swift_container_updater 95091d1645b9 satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1 kolla_start 25 hours ago Up 4 minutes ago swift_container_server 14f32cfad886 satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1 kolla_start 25 hours ago Up 6 minutes ago swift_account_server 355db438408f satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1 kolla_start 25 hours ago Up 3 minutes ago swift_account_reaper d8c959c6701f satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1 kolla_start 3 weeks ago Up 4 minutes ago nova_compute 82c8004e303a satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 4 minutes ago nova_api_cron a328cfc83542 satellite.example.local:5000/example_org-osp16_1_containers-ovn-northd:16.1 kolla_start 3 weeks ago Up 6 minutes ago ovn_northd 0e35a911d735 satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 5 minutes ago nova_metadata e3a9b4051325 satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 4 minutes ago nova_api 438b2329f777 satellite.example.local:5000/example_org-osp16_1_containers-glance-api:16.1 kolla_start 3 weeks ago Up 4 minutes ago glance_api 7452ed8b9438 satellite.example.local:5000/example_org-osp16_1_containers-placement-api:16.1 kolla_start 3 weeks ago Up 6 minutes ago placement_api 9a88e9d367f7 satellite.example.local:5000/example_org-osp16_1_containers-neutron-metadata-agent-ovn:16.1 kolla_start 3 weeks ago Up 4 minutes ago ovn_metadata_agent 96adba28368c satellite.example.local:5000/example_org-osp16_1_containers-ovn-controller:16.1 kolla_start 3 weeks ago Up 5 minutes ago ovn_controller ffcfbcf8b073 satellite.example.local:5000/example_org-osp16_1_containers-ovn-sb-db-server:16.1 kolla_start 3 weeks ago Up 4 minutes ago ovn_south_db_server 254807a79502 satellite.example.local:5000/example_org-osp16_1_containers-ovn-nb-db-server:16.1 kolla_start 3 weeks ago Up 3 minutes ago ovn_north_db_server 4fa033da4bc1 satellite.example.local:5000/example_org-osp16_1_containers-nova-novncproxy:16.1 kolla_start 3 weeks ago Up 5 minutes ago nova_vnc_proxy c05da9dc8d9c satellite.example.local:5000/example_org-osp16_1_containers-nova-scheduler:16.1 kolla_start 3 weeks ago Up 32 seconds ago nova_scheduler 921211859f02 satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1 kolla_start 3 weeks ago Up 6 minutes ago nova_migration_target e8e9d087d1c4 satellite.example.local:5000/example_org-osp16_1_containers-nova-conductor:16.1 kolla_start 3 weeks ago Up 17 seconds ago nova_conductor 5e6309c08256 satellite.example.local:5000/example_org-osp16_1_containers-neutron-server-ovn:16.1 kolla_start 3 weeks ago Up 6 minutes ago neutron_api 42d87756b0e5 satellite.example.local:5000/example_org-osp16_1_containers-cron:16.1 kolla_start 3 weeks ago Up 6 minutes ago logrotate_crond 46e61bbe2ff5 satellite.example.local:5000/example_org-osp16_1_containers-cinder-scheduler:16.1 kolla_start 3 weeks ago Up 6 minutes ago cinder_scheduler b7a5c18e9657 satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1 kolla_start 3 weeks ago Up 4 minutes ago cinder_api_cron 5b58b58f40f2 satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1 kolla_start 3 weeks ago Up 5 minutes ago cinder_api 5f14f438820c satellite.example.local:5000/example_org-osp16_1_containers-keystone:16.1 kolla_start 3 weeks ago Up 6 minutes ago keystone beee4ab76d7e satellite.example.local:5000/example_org-osp16_1_containers-iscsid:16.1 kolla_start 3 weeks ago Up 5 minutes ago iscsid 65a6c20d0636 satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1 kolla_start 3 weeks ago Up 4 minutes ago nova_libvirt 3a6961cd10d1 satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1 kolla_start 3 weeks ago Up 6 minutes ago nova_virtlogd 4d4ab1027038 satellite.example.local:5000/example_org-osp16_1_containers-horizon:16.1 kolla_start 3 weeks ago Up 6 minutes ago horizon 3ebcb1b4a278 satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1 kolla_start 3 weeks ago Up 4 minutes ago clustercheck 7e935ce984cf satellite.example.local:5000/example_org-osp16_1_containers-memcached:16.1 kolla_start 3 weeks ago Up 4 minutes ago memcached [stack@openstack16 ~]$ sudo pcs status Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum * Last updated: Thu May 6 21:08:22 2021 * Last change: Wed May 5 20:06:34 2021 by hacluster via crmd on openstack16 * 3 nodes configured * 7 resource instances configured Node List: * Online: [ openstack16 ] * GuestOnline: [ rabbitmq-bundle-0@openstack16 ] Full List of Resources: * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]: * galera-bundle-0 (ocf::heartbeat:galera): Stopped * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]: * rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started openstack16 * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]: * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Started openstack16 Failed Resource Actions: * galera-bundle-podman-0_start_0 on openstack16 'error' (1): call=14, status='Timed Out', exitreason='', last-rc-change='2021-05-06 20:59:13Z', queued=1ms, exec=120005ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@openstack16 ~]# grep -i galera /var/log/messages May 6 20:58:26 openstack16 pacemaker-controld[2077]: notice: Initiating monitor operation galera-bundle-podman-0_monitor_0 locally on openstack16 May 6 20:59:11 openstack16 pacemaker-controld[2077]: notice: Result of probe operation for galera-bundle-podman-0 on openstack16: 7 (not running) May 6 20:59:13 openstack16 pacemaker-controld[2077]: notice: Initiating start operation galera-bundle-podman-0_start_0 locally on openstack16 May 6 20:59:59 openstack16 podman(galera-bundle-podman-0)[2857]: INFO: running container galera-bundle-podman-0 for the first time May 6 21:01:11 openstack16 podman[4324]: 2021-05-06 21:01:11.39527107 +0000 UTC m=+71.520149007 container create f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0) May 6 21:01:13 openstack16 pacemaker-execd[2074]: warning: galera-bundle-podman-0_start_0 process (PID 2857) timed out May 6 21:01:13 openstack16 pacemaker-execd[2074]: warning: galera-bundle-podman-0_start_0:2857 - timed out after 120000ms May 6 21:01:13 openstack16 pacemaker-controld[2077]: error: Result of start operation for galera-bundle-podman-0 on openstack16: Timed Out May 6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Transition 0 aborted by operation galera-bundle-podman-0_start_0 'modify' on openstack16: Event failed May 6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Transition 0 action 6 (galera-bundle-podman-0_start_0 on openstack16): expected 'ok' but got 'error' May 6 21:01:13 openstack16 pacemaker-attrd[2075]: notice: Setting fail-count-galera-bundle-podman-0#start_0[openstack16]: (unset) -> INFINITY May 6 21:01:13 openstack16 pacemaker-attrd[2075]: notice: Setting last-failure-galera-bundle-podman-0#start_0[openstack16]: (unset) -> 1620334873 May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Fence (reboot) galera-bundle-0 (resource: galera-bundle-podman-0) 'guest is unclean' May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Recover galera-bundle-podman-0 ( openstack16 ) May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera-bundle-0 ( openstack16 ) May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera:0 ( galera-bundle-0 ) May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000) May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Fence (off) galera-bundle-0 (resource: galera-bundle-podman-0) 'guest is unclean' May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Stop galera-bundle-podman-0 ( openstack16 ) due to node availability May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera-bundle-0 ( openstack16 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera:0 ( galera-bundle-0 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Initiating monitor operation galera-bundle-0_monitor_0 locally on openstack16 May 6 21:01:14 openstack16 pacemaker-controld[2077]: notice: Result of probe operation for galera-bundle-0 on openstack16: 7 (not running) May 6 21:01:14 openstack16 pacemaker-controld[2077]: notice: Initiating stop operation galera-bundle-podman-0_stop_0 locally on openstack16 May 6 21:01:36 openstack16 podman[5792]: 2021-05-06 21:01:36.855131181 +0000 UTC m=+23.410980297 container cleanup f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0) May 6 21:01:42 openstack16 podman(galera-bundle-podman-0)[5808]: NOTICE: Cleaning up inactive container, galera-bundle-podman-0. May 6 21:02:12 openstack16 podman[6759]: 2021-05-06 21:02:12.839309734 +0000 UTC m=+30.057322199 container remove f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0) May 6 21:02:12 openstack16 podman(galera-bundle-podman-0)[5808]: INFO: f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c May 6 21:02:12 openstack16 pacemaker-controld[2077]: notice: Result of stop operation for galera-bundle-podman-0 on openstack16: 0 (ok) May 6 21:04:18 openstack16 pacemaker-controld[2077]: notice: Node galera-bundle-0 state is now lost May 6 21:04:18 openstack16 pacemaker-attrd[2075]: notice: Removing all galera-bundle-0 attributes for peer openstack16 May 6 21:04:18 openstack16 pacemaker-fenced[2073]: notice: Node galera-bundle-0 state is now lost May 6 21:04:18 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:04:18 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000) May 6 21:04:18 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera-bundle-0 ( openstack16 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:04:18 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera:0 ( galera-bundle-0 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:04:57 openstack16 pacemaker-controld[2077]: notice: Node galera-bundle-0 state is now lost May 6 21:04:57 openstack16 pacemaker-controld[2077]: warning: Stonith/shutdown of node galera-bundle-0 was not expected May 6 21:04:57 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:04:57 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000) May 6 21:04:57 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera-bundle-0 ( openstack16 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:04:57 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera:0 ( galera-bundle-0 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:04:58 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May 6 20:59:13 2021 May 6 21:04:58 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000) May 6 21:04:58 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera-bundle-0 ( openstack16 ) due to unrunnable galera-bundle-podman-0 start (blocked) May 6 21:04:58 openstack16 pacemaker-schedulerd[2076]: notice: * Start galera:0 ( galera-bundle-0 ) due to unrunnable galera-bundle-podman-0 start (blocked) === Try to run any of these generated playbooks available at the /home/stack/standalone-ansible-xxxxxxxx directory: $ sudo ansible-playbook -i inventory.yaml update_steps_playbook.yaml $ sudo ansible-playbook -i inventory.yaml deploy_steps_playbook.yaml One of them will succeed (maybe after trying twice). Then check the status of the services again and the status will be OK: === (alexon_admin) [stack@openstack16 ~]$ openstack service list +----------------------------------+-----------+--------------+ | ID | Name | Type | +----------------------------------+-----------+--------------+ | 292d539db9bf451e93fa0c5862c0231a | neutron | network | | 3896f1033265422dbe410707d1f8f8cf | cinderv3 | volumev3 | | 4ac2dd1ac9e648b6981fe8cf641e64b9 | keystone | identity | | 50f4c610d3b24857ae6343337b5c5ddb | placement | placement | | 8fe19156e2304e118f12d82fdb15f443 | swift | object-store | | a835752d13ff4320af0edc306b275d28 | nova | compute | | d6a1240690d345939d5229708e0493b2 | cinderv2 | volumev2 | | e734c193fcd6429a9269a3687f517c8c | glance | image | +----------------------------------+-----------+--------------+ (alexon_admin) [stack@openstack16 ~]$ sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b58a332178ab satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1 kolla_start 24 hours ago Up 24 hours ago swift_proxy 70b451a661f1 satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 24 hours ago Up 24 hours ago swift_rsync 0c07391fd692 satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 24 hours ago Up 24 hours ago swift_object_updater 97c91bd9ad6c satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1 kolla_start 24 hours ago Up 24 hours ago swift_object_server ca8a2a5a13ee satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1 kolla_start 24 hours ago Up 24 hours ago swift_object_expirer f54832f70f51 satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1 kolla_start 24 hours ago Up 24 hours ago swift_container_updater 95091d1645b9 satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1 kolla_start 24 hours ago Up 24 hours ago swift_container_server 14f32cfad886 satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1 kolla_start 24 hours ago Up 24 hours ago swift_account_server 355db438408f satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1 kolla_start 24 hours ago Up 24 hours ago swift_account_reaper 01847337997e satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1 /bin/bash /usr/lo... 25 hours ago Up 25 hours ago galera-bundle-podman-0 0cfa7c49dbd2 satellite.example.local:5000/example_org-osp16_1_containers-cinder-volume:16.1 /bin/bash /usr/lo... 25 hours ago Up 25 hours ago openstack-cinder-volume-podman-0 8594ddab837e satellite.example.local:5000/example_org-osp16_1_containers-rabbitmq:16.1 /bin/bash /usr/lo... 25 hours ago Up 25 hours ago rabbitmq-bundle-podman-0 d8c959c6701f satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_compute 82c8004e303a satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_api_cron a328cfc83542 satellite.example.local:5000/example_org-osp16_1_containers-ovn-northd:16.1 kolla_start 3 weeks ago Up 28 hours ago ovn_northd 0e35a911d735 satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_metadata e3a9b4051325 satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_api 438b2329f777 satellite.example.local:5000/example_org-osp16_1_containers-glance-api:16.1 kolla_start 3 weeks ago Up 28 hours ago glance_api 7452ed8b9438 satellite.example.local:5000/example_org-osp16_1_containers-placement-api:16.1 kolla_start 3 weeks ago Up 28 hours ago placement_api 9a88e9d367f7 satellite.example.local:5000/example_org-osp16_1_containers-neutron-metadata-agent-ovn:16.1 kolla_start 3 weeks ago Up 28 hours ago ovn_metadata_agent 96adba28368c satellite.example.local:5000/example_org-osp16_1_containers-ovn-controller:16.1 kolla_start 3 weeks ago Up 28 hours ago ovn_controller ffcfbcf8b073 satellite.example.local:5000/example_org-osp16_1_containers-ovn-sb-db-server:16.1 kolla_start 3 weeks ago Up 28 hours ago ovn_south_db_server 254807a79502 satellite.example.local:5000/example_org-osp16_1_containers-ovn-nb-db-server:16.1 kolla_start 3 weeks ago Up 28 hours ago ovn_north_db_server 4fa033da4bc1 satellite.example.local:5000/example_org-osp16_1_containers-nova-novncproxy:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_vnc_proxy c05da9dc8d9c satellite.example.local:5000/example_org-osp16_1_containers-nova-scheduler:16.1 kolla_start 3 weeks ago Up 25 hours ago nova_scheduler 921211859f02 satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_migration_target e8e9d087d1c4 satellite.example.local:5000/example_org-osp16_1_containers-nova-conductor:16.1 kolla_start 3 weeks ago Up 25 hours ago nova_conductor 5e6309c08256 satellite.example.local:5000/example_org-osp16_1_containers-neutron-server-ovn:16.1 kolla_start 3 weeks ago Up 28 hours ago neutron_api 42d87756b0e5 satellite.example.local:5000/example_org-osp16_1_containers-cron:16.1 kolla_start 3 weeks ago Up 28 hours ago logrotate_crond 46e61bbe2ff5 satellite.example.local:5000/example_org-osp16_1_containers-cinder-scheduler:16.1 kolla_start 3 weeks ago Up 28 hours ago cinder_scheduler b7a5c18e9657 satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1 kolla_start 3 weeks ago Up 28 hours ago cinder_api_cron 5b58b58f40f2 satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1 kolla_start 3 weeks ago Up 28 hours ago cinder_api 5f14f438820c satellite.example.local:5000/example_org-osp16_1_containers-keystone:16.1 kolla_start 3 weeks ago Up 28 hours ago keystone beee4ab76d7e satellite.example.local:5000/example_org-osp16_1_containers-iscsid:16.1 kolla_start 3 weeks ago Up 28 hours ago iscsid 65a6c20d0636 satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_libvirt 3a6961cd10d1 satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1 kolla_start 3 weeks ago Up 28 hours ago nova_virtlogd 4d4ab1027038 satellite.example.local:5000/example_org-osp16_1_containers-horizon:16.1 kolla_start 3 weeks ago Up 28 hours ago horizon 3ebcb1b4a278 satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1 kolla_start 3 weeks ago Up 28 hours ago clustercheck 7e935ce984cf satellite.example.local:5000/example_org-osp16_1_containers-memcached:16.1 kolla_start 3 weeks ago Up 28 hours ago memcached (alexon_admin) [stack@openstack16 ~]$ sudo pcs status Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum * Last updated: Thu May 6 20:53:08 2021 * Last change: Wed May 5 17:04:10 2021 by hacluster via crmd on openstack16 * 3 nodes configured * 7 resource instances configured Node List: * Online: [ openstack16 ] * GuestOnline: [ galera-bundle-0@openstack16 rabbitmq-bundle-0@openstack16 ] Full List of Resources: * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]: * galera-bundle-0 (ocf::heartbeat:galera): Master openstack16 * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]: * rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started openstack16 * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]: * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Started openstack16 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ===