Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1957979

Summary: Getting Gateway Timeout (HTTP 504) after restart the RHOSP 16.1 All-In-One server with default deployment
Product: Red Hat OpenStack Reporter: Alexon Oliveira <alolivei>
Component: openstack-tripleo-heat-templatesAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Joe H. Rahme <jhakimra>
Severity: low Docs Contact:
Priority: low    
Version: 16.1 (Train)CC: bdobreli, dabarzil, dciabrin, lmiccini, mbayer, mburns, rhos-maint
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-25 12:39:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexon Oliveira 2021-05-06 21:43:22 UTC
Description of problem:

On a fresh RHOSP 16.1 All-In-One default deployment (Tech Preview), every time the all-in-one server is rebooted, is not possible to authenticate with Keystone or run any openstack command due a "Gateway Timeout (HTTP 504)" error message:

$ openstack service list
Gateway Timeout (HTTP 504)

It seems like paunch is not being able to start some containers properly and pacemaker is failing to start galera-bundle resource either. Also, Horizon and Keystone do not seem to work at all. The only way to make it work properly again is to run the deploy_steps_playbook.yaml or the update_steps_playbook.yaml, but after run them once and reboot again, it does not work anymore.

Version-Release number of selected component (if applicable):

16.1

How reproducible:

Always

Steps to Reproduce:

1. Deploy the all-in-one solution with default options as per the documentation
2. Reboot the all-in-one server
3. Try to authenticate via CLI and run any openstack command. Horizon is not available either.

Actual results:

Fails (Services failing after reboot)

Expected results:

Succeeds (Services needs to work as usual after reboot)

Additional info:

Environment information:

===
[root@openstack16 ~]# rpm -qa | grep tripleo

openstack-tripleo-puppet-elements-11.2.2-1.20201114042506.f061f90.el8ost.noarch
python3-tripleoclient-heat-installer-12.3.2-1.20201114043247.1.el8ost.noarch
puppet-tripleo-11.5.0-1.20201114030109.el8ost.noarch
tripleo-ansible-0.5.1-1.20201114030848.2.el8ost.noarch
openstack-tripleo-common-containers-11.4.1-1.20210104173607.el8ost.noarch
openstack-tripleo-common-11.4.1-1.20210104173607.el8ost.noarch
python3-tripleoclient-12.3.2-1.20201114043247.1.el8ost.noarch
ansible-tripleo-ipsec-9.3.0-1.20201113193132.0c8693c.el8ost.noarch
ansible-role-tripleo-modify-image-1.2.1-1.20201114004656.1dffa21.el8ost.noarch
ansible-tripleo-ipa-0.2.1-1.20201114010033.3bb3c53.el8ost.noarch
python3-tripleo-common-11.4.1-1.20210104173607.el8ost.noarch
openstack-tripleo-validations-11.3.2-1.20201114040746.el8ost.noarch
openstack-tripleo-heat-templates-11.3.2-1.20210104205664.el8ost.2.noarch
openstack-tripleo-image-elements-10.6.2-1.20201113215051.7dc0fa1.el8ost.noarch

[root@openstack16 ~]# rpm -qa | grep openstack | grep -v tripleo

openstack-ironic-python-agent-builder-2.2.0-1.20201114043345.69e41ff.el8ost.noarch
python3-openstacksdk-0.36.4-1.20201113235937.76d3b29.el8ost.noarch
openstack-octavia-common-5.0.3-1.20201114024052.8c32d2e.el8ost.noarch
openstack-heat-monolith-13.1.0-1.20201114032442.48b730a.el8ost.noarch
openstack-octavia-diskimage-create-5.0.3-1.20201114024052.8c32d2e.el8ost.noarch
python3-openstackclient-4.0.1-1.20201114012609.bff556c.el8ost.noarch
puppet-openstacklib-15.4.1-1.20201113204514.5fdf43c.el8ost.noarch
python-openstackclient-lang-4.0.1-1.20201114012609.bff556c.el8ost.noarch
openstack-heat-common-13.1.0-1.20201114032442.48b730a.el8ost.noarch
openstack-heat-agents-1.10.1-1.20201113195131.96b819c.el8ost.noarch
openstack-selinux-0.8.24-1.20200914163012.26243bf.el8ost.noarch
ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.noarch
openstack-heat-api-13.1.0-1.20201114032442.48b730a.el8ost.noarch
puppet-openstack_extras-15.4.1-1.20201113215732.371931c.el8ost.noarch
openstack-heat-engine-13.1.0-1.20201114032442.48b730a.el8ost.noarch

[root@openstack16 ~]# hostnamectl 
   Static hostname: openstack16.example.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: f85aa0aaf6894ab792bd555197a13152
           Boot ID: 1023c63e1117489681daf0bd9595f29d
    Virtualization: kvm
  Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
            Kernel: Linux 4.18.0-193.47.1.el8_2.x86_64
      Architecture: x86-64
===

This is the status of a all-in-one server after a reboot:

===
[stack@openstack16 ~]$ source admin-openrc.sh
Please enter your OpenStack Password for project admin as user admin: 

[stack@openstack16 ~]$ openstack service list
Gateway Timeout (HTTP 504)

[stack@openstack16 ~]$ openstack service list
Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)

[stack@openstack16 ~]$ openstack --debug service list
START with options: --debug service list
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='http://192.168.3.2:5000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='default', project_domain_name='', project_id='65a612ee274b48fea0941df2da0dc549', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
messaging API version 2, cmd group openstack.messaging.v2
database API version 1, cmd group openstack.database.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
baremetal API version 1.58, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
event API version 2, cmd group openstack.event.v2
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
command: service list -> openstackclient.identity.v3.service.ListService (auth=True)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
Using auth plugin: password
Using parameters {'auth_url': 'http://192.168.3.2:5000', 'project_id': '65a612ee274b48fea0941df2da0dc549', 'project_name': 'admin', 'project_domain_id': 'default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.3.2:5000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.4 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8"
Starting new HTTP connection (1): 192.168.3.2:5000
http://192.168.3.2:5000 "GET / HTTP/1.1" 504 247
RESP: [504] Connection: Keep-Alive Content-Length: 247 Content-Type: text/html; charset=iso-8859-1 Date: Thu, 06 May 2021 21:26:45 GMT Keep-Alive: timeout=15, max=100 Server: Apache
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
Request returned failure status: 504
Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
    authenticated=False)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
    disc = Discover(session, url, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
clean_up ListService: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
    authenticated=False)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
    disc = Discover(session, url, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)

END return value: 1

[stack@openstack16 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                        COMMAND               CREATED        STATUS             PORTS  NAMES
66478086daed  satellite.example.local:5000/example_org-osp16_1_containers-cinder-volume:16.1               /bin/bash /usr/lo...  3 minutes ago  Up 3 minutes ago          openstack-cinder-volume-podman-0
5c9f5c9b5f12  satellite.example.local:5000/example_org-osp16_1_containers-rabbitmq:16.1                    /bin/bash /usr/lo...  5 minutes ago  Up 4 minutes ago          rabbitmq-bundle-podman-0
b58a332178ab  satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1          kolla_start           25 hours ago   Up 5 minutes ago          swift_proxy
70b451a661f1  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           25 hours ago   Up 3 minutes ago          swift_rsync
0c07391fd692  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           25 hours ago   Up 4 minutes ago          swift_object_updater
97c91bd9ad6c  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           25 hours ago   Up 4 minutes ago          swift_object_server
ca8a2a5a13ee  satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1          kolla_start           25 hours ago   Up 5 minutes ago          swift_object_expirer
f54832f70f51  satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1             kolla_start           25 hours ago   Up 4 minutes ago          swift_container_updater
95091d1645b9  satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1             kolla_start           25 hours ago   Up 4 minutes ago          swift_container_server
14f32cfad886  satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1               kolla_start           25 hours ago   Up 6 minutes ago          swift_account_server
355db438408f  satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1               kolla_start           25 hours ago   Up 3 minutes ago          swift_account_reaper
d8c959c6701f  satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1                kolla_start           3 weeks ago    Up 4 minutes ago          nova_compute
82c8004e303a  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago    Up 4 minutes ago          nova_api_cron
a328cfc83542  satellite.example.local:5000/example_org-osp16_1_containers-ovn-northd:16.1                  kolla_start           3 weeks ago    Up 6 minutes ago          ovn_northd
0e35a911d735  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago    Up 5 minutes ago          nova_metadata
e3a9b4051325  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago    Up 4 minutes ago          nova_api
438b2329f777  satellite.example.local:5000/example_org-osp16_1_containers-glance-api:16.1                  kolla_start           3 weeks ago    Up 4 minutes ago          glance_api
7452ed8b9438  satellite.example.local:5000/example_org-osp16_1_containers-placement-api:16.1               kolla_start           3 weeks ago    Up 6 minutes ago          placement_api
9a88e9d367f7  satellite.example.local:5000/example_org-osp16_1_containers-neutron-metadata-agent-ovn:16.1  kolla_start           3 weeks ago    Up 4 minutes ago          ovn_metadata_agent
96adba28368c  satellite.example.local:5000/example_org-osp16_1_containers-ovn-controller:16.1              kolla_start           3 weeks ago    Up 5 minutes ago          ovn_controller
ffcfbcf8b073  satellite.example.local:5000/example_org-osp16_1_containers-ovn-sb-db-server:16.1            kolla_start           3 weeks ago    Up 4 minutes ago          ovn_south_db_server
254807a79502  satellite.example.local:5000/example_org-osp16_1_containers-ovn-nb-db-server:16.1            kolla_start           3 weeks ago    Up 3 minutes ago          ovn_north_db_server
4fa033da4bc1  satellite.example.local:5000/example_org-osp16_1_containers-nova-novncproxy:16.1             kolla_start           3 weeks ago    Up 5 minutes ago          nova_vnc_proxy
c05da9dc8d9c  satellite.example.local:5000/example_org-osp16_1_containers-nova-scheduler:16.1              kolla_start           3 weeks ago    Up 32 seconds ago         nova_scheduler
921211859f02  satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1                kolla_start           3 weeks ago    Up 6 minutes ago          nova_migration_target
e8e9d087d1c4  satellite.example.local:5000/example_org-osp16_1_containers-nova-conductor:16.1              kolla_start           3 weeks ago    Up 17 seconds ago         nova_conductor
5e6309c08256  satellite.example.local:5000/example_org-osp16_1_containers-neutron-server-ovn:16.1          kolla_start           3 weeks ago    Up 6 minutes ago          neutron_api
42d87756b0e5  satellite.example.local:5000/example_org-osp16_1_containers-cron:16.1                        kolla_start           3 weeks ago    Up 6 minutes ago          logrotate_crond
46e61bbe2ff5  satellite.example.local:5000/example_org-osp16_1_containers-cinder-scheduler:16.1            kolla_start           3 weeks ago    Up 6 minutes ago          cinder_scheduler
b7a5c18e9657  satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1                  kolla_start           3 weeks ago    Up 4 minutes ago          cinder_api_cron
5b58b58f40f2  satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1                  kolla_start           3 weeks ago    Up 5 minutes ago          cinder_api
5f14f438820c  satellite.example.local:5000/example_org-osp16_1_containers-keystone:16.1                    kolla_start           3 weeks ago    Up 6 minutes ago          keystone
beee4ab76d7e  satellite.example.local:5000/example_org-osp16_1_containers-iscsid:16.1                      kolla_start           3 weeks ago    Up 5 minutes ago          iscsid
65a6c20d0636  satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1                kolla_start           3 weeks ago    Up 4 minutes ago          nova_libvirt
3a6961cd10d1  satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1                kolla_start           3 weeks ago    Up 6 minutes ago          nova_virtlogd
4d4ab1027038  satellite.example.local:5000/example_org-osp16_1_containers-horizon:16.1                     kolla_start           3 weeks ago    Up 6 minutes ago          horizon
3ebcb1b4a278  satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1                     kolla_start           3 weeks ago    Up 4 minutes ago          clustercheck
7e935ce984cf  satellite.example.local:5000/example_org-osp16_1_containers-memcached:16.1                   kolla_start           3 weeks ago    Up 4 minutes ago          memcached

[stack@openstack16 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum
  * Last updated: Thu May  6 21:08:22 2021
  * Last change:  Wed May  5 20:06:34 2021 by hacluster via crmd on openstack16
  * 3 nodes configured
  * 7 resource instances configured

Node List:
  * Online: [ openstack16 ]
  * GuestOnline: [ rabbitmq-bundle-0@openstack16 ]

Full List of Resources:
  * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
    * galera-bundle-0	(ocf::heartbeat:galera):	Stopped
  * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
    * rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started openstack16
  * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
    * openstack-cinder-volume-podman-0	(ocf::heartbeat:podman):	Started openstack16

Failed Resource Actions:
  * galera-bundle-podman-0_start_0 on openstack16 'error' (1): call=14, status='Timed Out', exitreason='', last-rc-change='2021-05-06 20:59:13Z', queued=1ms, exec=120005ms

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

[root@openstack16 ~]# grep -i galera /var/log/messages

May  6 20:58:26 openstack16 pacemaker-controld[2077]: notice: Initiating monitor operation galera-bundle-podman-0_monitor_0 locally on openstack16
May  6 20:59:11 openstack16 pacemaker-controld[2077]: notice: Result of probe operation for galera-bundle-podman-0 on openstack16: 7 (not running)
May  6 20:59:13 openstack16 pacemaker-controld[2077]: notice: Initiating start operation galera-bundle-podman-0_start_0 locally on openstack16
May  6 20:59:59 openstack16 podman(galera-bundle-podman-0)[2857]: INFO: running container galera-bundle-podman-0 for the first time
May  6 21:01:11 openstack16 podman[4324]: 2021-05-06 21:01:11.39527107 +0000 UTC m=+71.520149007 container create f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0)
May  6 21:01:13 openstack16 pacemaker-execd[2074]: warning: galera-bundle-podman-0_start_0 process (PID 2857) timed out
May  6 21:01:13 openstack16 pacemaker-execd[2074]: warning: galera-bundle-podman-0_start_0:2857 - timed out after 120000ms
May  6 21:01:13 openstack16 pacemaker-controld[2077]: error: Result of start operation for galera-bundle-podman-0 on openstack16: Timed Out
May  6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Transition 0 aborted by operation galera-bundle-podman-0_start_0 'modify' on openstack16: Event failed
May  6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Transition 0 action 6 (galera-bundle-podman-0_start_0 on openstack16): expected 'ok' but got 'error'
May  6 21:01:13 openstack16 pacemaker-attrd[2075]: notice: Setting fail-count-galera-bundle-podman-0#start_0[openstack16]: (unset) -> INFINITY
May  6 21:01:13 openstack16 pacemaker-attrd[2075]: notice: Setting last-failure-galera-bundle-podman-0#start_0[openstack16]: (unset) -> 1620334873
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Fence (reboot) galera-bundle-0 (resource: galera-bundle-podman-0) 'guest is unclean'
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Recover    galera-bundle-podman-0               (       openstack16 )
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera-bundle-0                      (       openstack16 )
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera:0                             (   galera-bundle-0 )
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000)
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Fence (off) galera-bundle-0 (resource: galera-bundle-podman-0) 'guest is unclean'
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Stop       galera-bundle-podman-0               (       openstack16 )   due to node availability
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera-bundle-0                      (       openstack16 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:01:13 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera:0                             (   galera-bundle-0 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:01:13 openstack16 pacemaker-controld[2077]: notice: Initiating monitor operation galera-bundle-0_monitor_0 locally on openstack16
May  6 21:01:14 openstack16 pacemaker-controld[2077]: notice: Result of probe operation for galera-bundle-0 on openstack16: 7 (not running)
May  6 21:01:14 openstack16 pacemaker-controld[2077]: notice: Initiating stop operation galera-bundle-podman-0_stop_0 locally on openstack16
May  6 21:01:36 openstack16 podman[5792]: 2021-05-06 21:01:36.855131181 +0000 UTC m=+23.410980297 container cleanup f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0)
May  6 21:01:42 openstack16 podman(galera-bundle-podman-0)[5808]: NOTICE: Cleaning up inactive container, galera-bundle-podman-0.
May  6 21:02:12 openstack16 podman[6759]: 2021-05-06 21:02:12.839309734 +0000 UTC m=+30.057322199 container remove f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c (image=satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1, name=galera-bundle-podman-0)
May  6 21:02:12 openstack16 podman(galera-bundle-podman-0)[5808]: INFO: f47e41591b71e74f77fb17c37ab18606a6aee10ea77dc5904911bc0b5a3bc79c
May  6 21:02:12 openstack16 pacemaker-controld[2077]: notice: Result of stop operation for galera-bundle-podman-0 on openstack16: 0 (ok)
May  6 21:04:18 openstack16 pacemaker-controld[2077]: notice: Node galera-bundle-0 state is now lost
May  6 21:04:18 openstack16 pacemaker-attrd[2075]: notice: Removing all galera-bundle-0 attributes for peer openstack16
May  6 21:04:18 openstack16 pacemaker-fenced[2073]: notice: Node galera-bundle-0 state is now lost
May  6 21:04:18 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:04:18 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000)
May  6 21:04:18 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera-bundle-0                      (       openstack16 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:04:18 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera:0                             (   galera-bundle-0 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:04:57 openstack16 pacemaker-controld[2077]: notice: Node galera-bundle-0 state is now lost
May  6 21:04:57 openstack16 pacemaker-controld[2077]: warning: Stonith/shutdown of node galera-bundle-0 was not expected
May  6 21:04:57 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:04:57 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000)
May  6 21:04:57 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera-bundle-0                      (       openstack16 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:04:57 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera:0                             (   galera-bundle-0 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:04:58 openstack16 pacemaker-schedulerd[2076]: warning: Unexpected result (error) was recorded for start of galera-bundle-podman-0 on openstack16 at May  6 20:59:13 2021
May  6 21:04:58 openstack16 pacemaker-schedulerd[2076]: warning: Forcing galera-bundle-podman-0 away from openstack16 after 1000000 failures (max=1000000)
May  6 21:04:58 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera-bundle-0                      (       openstack16 )   due to unrunnable galera-bundle-podman-0 start (blocked)
May  6 21:04:58 openstack16 pacemaker-schedulerd[2076]: notice:  * Start      galera:0                             (   galera-bundle-0 )   due to unrunnable galera-bundle-podman-0 start (blocked)
===

Try to run any of these generated playbooks available at the /home/stack/standalone-ansible-xxxxxxxx directory:

$ sudo ansible-playbook -i inventory.yaml update_steps_playbook.yaml
$ sudo ansible-playbook -i inventory.yaml deploy_steps_playbook.yaml

One of them will succeed (maybe after trying twice). Then check the status of the services again and the status will be OK:

===
(alexon_admin) [stack@openstack16 ~]$ openstack service list
+----------------------------------+-----------+--------------+
| ID                               | Name      | Type         |
+----------------------------------+-----------+--------------+
| 292d539db9bf451e93fa0c5862c0231a | neutron   | network      |
| 3896f1033265422dbe410707d1f8f8cf | cinderv3  | volumev3     |
| 4ac2dd1ac9e648b6981fe8cf641e64b9 | keystone  | identity     |
| 50f4c610d3b24857ae6343337b5c5ddb | placement | placement    |
| 8fe19156e2304e118f12d82fdb15f443 | swift     | object-store |
| a835752d13ff4320af0edc306b275d28 | nova      | compute      |
| d6a1240690d345939d5229708e0493b2 | cinderv2  | volumev2     |
| e734c193fcd6429a9269a3687f517c8c | glance    | image        |
+----------------------------------+-----------+--------------+


(alexon_admin) [stack@openstack16 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                                        COMMAND               CREATED       STATUS           PORTS  NAMES
b58a332178ab  satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1          kolla_start           24 hours ago  Up 24 hours ago         swift_proxy
70b451a661f1  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           24 hours ago  Up 24 hours ago         swift_rsync
0c07391fd692  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           24 hours ago  Up 24 hours ago         swift_object_updater
97c91bd9ad6c  satellite.example.local:5000/example_org-osp16_1_containers-swift-object:16.1                kolla_start           24 hours ago  Up 24 hours ago         swift_object_server
ca8a2a5a13ee  satellite.example.local:5000/example_org-osp16_1_containers-swift-proxy-server:16.1          kolla_start           24 hours ago  Up 24 hours ago         swift_object_expirer
f54832f70f51  satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1             kolla_start           24 hours ago  Up 24 hours ago         swift_container_updater
95091d1645b9  satellite.example.local:5000/example_org-osp16_1_containers-swift-container:16.1             kolla_start           24 hours ago  Up 24 hours ago         swift_container_server
14f32cfad886  satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1               kolla_start           24 hours ago  Up 24 hours ago         swift_account_server
355db438408f  satellite.example.local:5000/example_org-osp16_1_containers-swift-account:16.1               kolla_start           24 hours ago  Up 24 hours ago         swift_account_reaper
01847337997e  satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1                     /bin/bash /usr/lo...  25 hours ago  Up 25 hours ago         galera-bundle-podman-0
0cfa7c49dbd2  satellite.example.local:5000/example_org-osp16_1_containers-cinder-volume:16.1               /bin/bash /usr/lo...  25 hours ago  Up 25 hours ago         openstack-cinder-volume-podman-0
8594ddab837e  satellite.example.local:5000/example_org-osp16_1_containers-rabbitmq:16.1                    /bin/bash /usr/lo...  25 hours ago  Up 25 hours ago         rabbitmq-bundle-podman-0
d8c959c6701f  satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1                kolla_start           3 weeks ago   Up 28 hours ago         nova_compute
82c8004e303a  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago   Up 28 hours ago         nova_api_cron
a328cfc83542  satellite.example.local:5000/example_org-osp16_1_containers-ovn-northd:16.1                  kolla_start           3 weeks ago   Up 28 hours ago         ovn_northd
0e35a911d735  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago   Up 28 hours ago         nova_metadata
e3a9b4051325  satellite.example.local:5000/example_org-osp16_1_containers-nova-api:16.1                    kolla_start           3 weeks ago   Up 28 hours ago         nova_api
438b2329f777  satellite.example.local:5000/example_org-osp16_1_containers-glance-api:16.1                  kolla_start           3 weeks ago   Up 28 hours ago         glance_api
7452ed8b9438  satellite.example.local:5000/example_org-osp16_1_containers-placement-api:16.1               kolla_start           3 weeks ago   Up 28 hours ago         placement_api
9a88e9d367f7  satellite.example.local:5000/example_org-osp16_1_containers-neutron-metadata-agent-ovn:16.1  kolla_start           3 weeks ago   Up 28 hours ago         ovn_metadata_agent
96adba28368c  satellite.example.local:5000/example_org-osp16_1_containers-ovn-controller:16.1              kolla_start           3 weeks ago   Up 28 hours ago         ovn_controller
ffcfbcf8b073  satellite.example.local:5000/example_org-osp16_1_containers-ovn-sb-db-server:16.1            kolla_start           3 weeks ago   Up 28 hours ago         ovn_south_db_server
254807a79502  satellite.example.local:5000/example_org-osp16_1_containers-ovn-nb-db-server:16.1            kolla_start           3 weeks ago   Up 28 hours ago         ovn_north_db_server
4fa033da4bc1  satellite.example.local:5000/example_org-osp16_1_containers-nova-novncproxy:16.1             kolla_start           3 weeks ago   Up 28 hours ago         nova_vnc_proxy
c05da9dc8d9c  satellite.example.local:5000/example_org-osp16_1_containers-nova-scheduler:16.1              kolla_start           3 weeks ago   Up 25 hours ago         nova_scheduler
921211859f02  satellite.example.local:5000/example_org-osp16_1_containers-nova-compute:16.1                kolla_start           3 weeks ago   Up 28 hours ago         nova_migration_target
e8e9d087d1c4  satellite.example.local:5000/example_org-osp16_1_containers-nova-conductor:16.1              kolla_start           3 weeks ago   Up 25 hours ago         nova_conductor
5e6309c08256  satellite.example.local:5000/example_org-osp16_1_containers-neutron-server-ovn:16.1          kolla_start           3 weeks ago   Up 28 hours ago         neutron_api
42d87756b0e5  satellite.example.local:5000/example_org-osp16_1_containers-cron:16.1                        kolla_start           3 weeks ago   Up 28 hours ago         logrotate_crond
46e61bbe2ff5  satellite.example.local:5000/example_org-osp16_1_containers-cinder-scheduler:16.1            kolla_start           3 weeks ago   Up 28 hours ago         cinder_scheduler
b7a5c18e9657  satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1                  kolla_start           3 weeks ago   Up 28 hours ago         cinder_api_cron
5b58b58f40f2  satellite.example.local:5000/example_org-osp16_1_containers-cinder-api:16.1                  kolla_start           3 weeks ago   Up 28 hours ago         cinder_api
5f14f438820c  satellite.example.local:5000/example_org-osp16_1_containers-keystone:16.1                    kolla_start           3 weeks ago   Up 28 hours ago         keystone
beee4ab76d7e  satellite.example.local:5000/example_org-osp16_1_containers-iscsid:16.1                      kolla_start           3 weeks ago   Up 28 hours ago         iscsid
65a6c20d0636  satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1                kolla_start           3 weeks ago   Up 28 hours ago         nova_libvirt
3a6961cd10d1  satellite.example.local:5000/example_org-osp16_1_containers-nova-libvirt:16.1                kolla_start           3 weeks ago   Up 28 hours ago         nova_virtlogd
4d4ab1027038  satellite.example.local:5000/example_org-osp16_1_containers-horizon:16.1                     kolla_start           3 weeks ago   Up 28 hours ago         horizon
3ebcb1b4a278  satellite.example.local:5000/example_org-osp16_1_containers-mariadb:16.1                     kolla_start           3 weeks ago   Up 28 hours ago         clustercheck
7e935ce984cf  satellite.example.local:5000/example_org-osp16_1_containers-memcached:16.1                   kolla_start           3 weeks ago   Up 28 hours ago         memcached

(alexon_admin) [stack@openstack16 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum
  * Last updated: Thu May  6 20:53:08 2021
  * Last change:  Wed May  5 17:04:10 2021 by hacluster via crmd on openstack16
  * 3 nodes configured
  * 7 resource instances configured

Node List:
  * Online: [ openstack16 ]
  * GuestOnline: [ galera-bundle-0@openstack16 rabbitmq-bundle-0@openstack16 ]

Full List of Resources:
  * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
    * galera-bundle-0	(ocf::heartbeat:galera):	Master openstack16
  * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
    * rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started openstack16
  * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
    * openstack-cinder-volume-podman-0	(ocf::heartbeat:podman):	Started openstack16

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
===

Comment 1 Luca Miccini 2021-05-07 13:20:07 UTC
Hi,

Judging from the output of pcs status there are at least a couple of things that are out of shape:

1. lack of "/usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml" so you don't have all the pacemaker resources defined properly
2. missing the control plane VIP.


Regarding 1. : this is automatically included when you deploy a overcloud via the plan-environment.yaml file and I guess the standalone hasn't been adjusted to reflect this.
Regarding 2. : this should be passed as an option to the deploy command


Can you try adjusting your deployment command to look something like this:

sudo openstack tripleo deploy --templates --local-ip=IP/NETMASK --control-virtual-ip=192.168.24.3 \
 -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
 -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
....

Please let us know if that helps.

Comment 2 Alexon Oliveira 2021-05-10 18:43:32 UTC
Hi Luca,

Thanks for the heads up. Just FYI, the deployment commands I ran was accordingly to the chapter 5 of the all-in-one 16.1 deployment official doc [1], like this:

===
[stack@openstack16 ~]$ cat deploy.sh
#!/bin/bash

sudo openstack tripleo deploy --templates --local-ip=$IP/$NETMASK \
-e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
-e $HOME/containers-prepare-parameters.yaml \
-e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone
===

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/standalone_deployment_guide/deploying-the-all-in-one-openstack-installation

I've changed it as you asked me to do, like the following:

===
[stack@openstack16 ~]$ cat deploy.sh
#!/bin/bash

sudo openstack tripleo deploy --templates --control-virtual-ip=$IP --local-ip=$IP/$NETMASK \
-e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
-e $HOME/containers-prepare-parameters.yaml \
-e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone
===

I ran a fresh newly deploy again with the adjusted deploy.sh file with the commands you've recommended and it ran with success, as follows:

===output ommited====
PLAY RECAP ******************************************************************************************************************************************
openstack16                : ok=341  changed=146  unreachable=0    failed=0    skipped=125  rescued=0    ignored=0
undercloud                 : ok=84   changed=30   unreachable=0    failed=0    skipped=11   rescued=0    ignored=0

Monday 10 May 2021  16:36:22 +0000 (0:00:00.063)       0:28:57.346 ************
===============================================================================
Wait for container-puppet tasks (generate config) to finish -------------------------------------------------------------------------------- 350.63s
Wait for containers to start for step 4 using paunch --------------------------------------------------------------------------------------- 254.35s
Wait for containers to start for step 2 using paunch --------------------------------------------------------------------------------------- 180.20s
Wait for containers to start for step 3 using paunch --------------------------------------------------------------------------------------- 120.67s
Wait for containers to start for step 5 using paunch --------------------------------------------------------------------------------------- 112.23s
Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 58.09s
Wait for container-puppet tasks (bootstrap tasks) for step 4 to finish ---------------------------------------------------------------------- 54.18s
Manage Cinder Volume Type ------------------------------------------------------------------------------------------------------------------- 30.98s
Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 29.54s
Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 27.50s
Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 24.86s
Run puppet on the host to apply IPtables rules ---------------------------------------------------------------------------------------------- 20.00s
Wait for container-puppet tasks (bootstrap tasks) for step 3 to finish ---------------------------------------------------------------------- 17.80s
Wait for puppet host configuration to finish ------------------------------------------------------------------------------------------------ 16.76s
Pre-fetch all the containers ----------------------------------------------------------------------------------------------------------------- 9.15s
tripleo-bootstrap : Deploy required packages to bootstrap TripleO ---------------------------------------------------------------------------- 7.94s
tripleo-keystone-resources : Check Keystone service status ----------------------------------------------------------------------------------- 7.42s
tripleo-keystone-resources : Check Keystone user assignment to roles status ------------------------------------------------------------------ 7.33s
tripleo-keystone-resources : Check Keystone internal endpoint status ------------------------------------------------------------------------- 6.97s
tripleo-keystone-resources : Check Keystone admin endpoint status ---------------------------------------------------------------------------- 6.91s
Not cleaning working directory /home/stack/tripleo-heat-installer-templates
Not cleaning ansible directory /home/stack/standalone-ansible-swk7qpid
Install artifact is located at /home/stack/standalone-install-20210510163622.tar.bzip2

########################################################

Deployment successful!

########################################################

##########################################################

Useful files:

The clouds.yaml file is at ~/.config/openstack/clouds.yaml

Use "export OS_CLOUD=standalone" before running the
openstack command.

##########################################################

Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_standalone
===

I managed to run any openstack command with no problem:

===
[stack@openstack16 ~]$ openstack service list
+----------------------------------+-----------+--------------+
| ID                               | Name      | Type         |
+----------------------------------+-----------+--------------+
| 2f34a4a42a5848c8b46cc5ab477481c8 | nova      | compute      |
| 541217a20275434ba6bf24e5eda7cc1d | glance    | image        |
| 6f598bf1c7174fb4afe96188d34d46ee | swift     | object-store |
| 797c9cafca4e495482e935388fa9dada | cinderv3  | volumev3     |
| 7be5974cd6864dfeaac622391302f975 | keystone  | identity     |
| 993209dadd1c4385970345d8168b56ec | neutron   | network      |
| c324eaa0ef40471b889acfe77740884f | cinderv2  | volumev2     |
| f8807878538649e381796ae7126adba6 | placement | placement    |
+----------------------------------+-----------+--------------+

[stack@openstack16 ~]$ sudo pcs cluster status
Cluster Status:
 Cluster Summary:
   * Stack: corosync
   * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum
   * Last updated: Mon May 10 18:21:22 2021
   * Last change:  Mon May 10 16:35:14 2021 by root via cibadmin on openstack16
   * 5 nodes configured
   * 15 resource instances configured
 Node List:
   * Online: [ openstack16 ]
   * GuestOnline: [ galera-bundle-0@openstack16 ovn-dbs-bundle-0@openstack16 rabbitmq-bundle-0@openstack16 redis-bundle-0@openstack16 ]

PCSD Status:
  openstack16: Online

[stack@openstack16 ~]$ uptime
 18:21:58 up  4:54,  1 user,  load average: 1,70, 1,50, 1,49
===

Then I did the test of reboot the all-in-one server and try to run any openstack command again, but as you can see below apparently only the pcs resource for galera-bundle has been solved, but not the fact I can't run any openstack command after reboot:

===
[stack@openstack16 ~]$ uptime
 18:29:44 up 4 min,  1 user,  load average: 12,83, 7,58, 3,18

[stack@openstack16 ~]$ sudo pcs cluster status
Cluster Status:
 Cluster Summary:
   * Stack: corosync
   * Current DC: openstack16 (version 2.0.3-5.el8_2.3-4b1f869f0f) - partition with quorum
   * Last updated: Mon May 10 18:29:54 2021
   * Last change:  Mon May 10 16:35:14 2021 by root via cibadmin on openstack16
   * 5 nodes configured
   * 15 resource instances configured
 Node List:
   * Online: [ openstack16 ]

PCSD Status:
  openstack16: Online

[stack@openstack16 ~]$ openstack service list
Gateway Timeout (HTTP 504)

[stack@openstack16 ~]$ openstack --debug service list
START with options: --debug service list
options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', aodh_endpoint='', application_credential_id='', application_credential_name='', application_credential_secret='***', auth_methods='', auth_type='', auth_url='http://192.168.3.2:5000', cacert=None, cert='', client_id='', client_secret='***', cloud='', code='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='public', key='', log_file=None, openid_scope='', os_alarming_api_version='2', os_baremetal_api_version='1.58', os_beta_command=False, os_compute_api_version='', os_container_infra_api_version='1', os_data_processing_api_version='1.1', os_data_processing_url='', os_database_api_version='1', os_dns_api_version='2', os_event_api_version='2', os_identity_api_version='3', os_image_api_version='', os_key_manager_api_version='1', os_loadbalancer_api_version='2.0', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile='', project_domain_id='default', project_domain_name='', project_id='08f36e84994f4e098abd115466d79b9c', project_name='admin', protocol='', redirect_uri='', region_name='regionOne', remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', remote_project_name='', roles='', service_provider='', service_provider_endpoint='', service_provider_entity_id='', system_scope='', timing=False, token='***', trust_id='', user='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
defaults: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'auth_type': 'password', 'baremetal_status_code_retries': 5, 'baremetal_introspection_status_code_retries': 5, 'image_status_code_retries': 5, 'disable_vendor_agent': {}, 'interface': None, 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active'}
cloud cfg: {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
compute API version 2.1, cmd group openstack.compute.v2
identity API version 3, cmd group openstack.identity.v3
image API version 2, cmd group openstack.image.v2
network API version 2, cmd group openstack.network.v2
object_store API version 1, cmd group openstack.object_store.v1
volume API version 3, cmd group openstack.volume.v3
messaging API version 2, cmd group openstack.messaging.v2
database API version 1, cmd group openstack.database.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
data_processing API version 1.1, cmd group openstack.data_processing.v1
load_balancer API version 2.0, cmd group openstack.load_balancer.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
container_infra API version 1, cmd group openstack.container_infra.v1
baremetal API version 1.58, cmd group openstack.baremetal.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
orchestration API version 1, cmd group openstack.orchestration.v1
dns API version 2, cmd group openstack.dns.v2
key_manager API version 1, cmd group openstack.key_manager.v1
event API version 2, cmd group openstack.event.v2
metric API version 1, cmd group openstack.metric.v1
alarming API version 2, cmd group openstack.alarming.v2
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
command: service list -> openstackclient.identity.v3.service.ListService (auth=True)
Auth plugin password selected
auth_config_hook(): {'api_timeout': None, 'verify': True, 'cacert': None, 'cert': None, 'key': None, 'baremetal_status_code_retries': '5', 'baremetal_introspection_status_code_retries': '5', 'image_status_code_retries': '5', 'disable_vendor_agent': {}, 'interface': 'public', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'image_format': 'qcow2', 'message': '', 'network_api_version': '2', 'object_store_api_version': '1', 'secgroup_source': 'neutron', 'status': 'active', 'auth': {'user_domain_name': 'Default', 'project_domain_id': 'default', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin'}, 'additional_user_agent': [('osc-lib', '1.14.1')], 'verbose_level': 3, 'deferred_help': False, 'debug': True, 'region_name': 'regionOne', 'default_domain': 'default', 'timing': False, 'inspector_api_version': '1', 'auth_url': 'http://192.168.3.2:5000', 'username': 'admin', 'password': '***', 'beta_command': False, 'identity_api_version': '3', 'queues_api_version': '2', 'database_api_version': '1', 'tripleoclient_api_version': '1', 'data_processing_api_version': '1.1', 'loadbalancer_api_version': '2.0', 'workflow_api_version': '2', 'container_infra_api_version': '1', 'baremetal_api_version': '1.58', 'orchestration_api_version': '1', 'dns_api_version': '2', 'key_manager_api_version': '1', 'event_api_version': '2', 'metrics_api_version': '1', 'alarming_api_version': '2', 'auth_type': 'password', 'networks': []}
Using auth plugin: password
Using parameters {'auth_url': 'http://192.168.3.2:5000', 'project_id': '08f36e84994f4e098abd115466d79b9c', 'project_name': 'admin', 'project_domain_id': 'default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'}
Get auth_ref
REQ: curl -g -i -X GET http://192.168.3.2:5000 -H "Accept: application/json" -H "User-Agent: openstacksdk/0.36.4 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8"
Starting new HTTP connection (1): 192.168.3.2:5000
http://192.168.3.2:5000 "GET / HTTP/1.1" 504 247
RESP: [504] Connection: Keep-Alive Content-Length: 247 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 10 May 2021 18:32:35 GMT Keep-Alive: timeout=15, max=100 Server: Apache
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
Request returned failure status: 504
Failed to discover available identity versions when contacting http://192.168.3.2:5000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
    authenticated=False)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
    disc = Discover(session, url, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
clean_up ListService: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
    authenticated=False)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
    disc = Discover(session, url, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
    authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1116, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 961, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 394, in run_subcommand
    self.prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/openstackclient/shell.py", line 166, in prepare_to_run_command
    return super(OpenStackShell, self).prepare_to_run_command(cmd)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 493, in prepare_to_run_command
    self.client_manager.auth_ref
  File "/usr/lib/python3.6/site-packages/osc_lib/clientmanager.py", line 202, in auth_ref
    self._auth_ref = self.auth.get_auth_ref(self.session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Gateway Timeout (HTTP 504)

END return value: 1

[stack@openstack16 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                                     COMMAND               CREATED        STATUS             PORTS  NAMES
89f275708376  registry.redhat.io/rhosp-rhel8/openstack-cinder-volume:16.1               /bin/bash /usr/lo...  2 minutes ago  Up 2 minutes ago          openstack-cinder-volume-podman-0
5ba1b5741373  registry.redhat.io/rhosp-rhel8/openstack-ovn-northd:16.1                  /bin/bash /usr/lo...  2 minutes ago  Up 2 minutes ago          ovn-dbs-bundle-podman-0
e5b4340380f2  registry.redhat.io/rhosp-rhel8/openstack-redis:16.1                       /bin/bash /usr/lo...  3 minutes ago  Up 3 minutes ago          redis-bundle-podman-0
1b94c194a443  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.1                    /bin/bash /usr/lo...  4 minutes ago  Up 3 minutes ago          rabbitmq-bundle-podman-0
21de845dbd17  registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.1                kolla_start           2 hours ago    Up 3 minutes ago          nova_compute
c3449d54f8f1  registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1                    kolla_start           2 hours ago    Up 4 minutes ago          nova_api_cron
a84ac13f19bb  registry.redhat.io/rhosp-rhel8/openstack-swift-proxy-server:16.1          kolla_start           2 hours ago    Up 4 minutes ago          swift_proxy
6a493cab06f7  registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1                    kolla_start           2 hours ago    Up 4 minutes ago          nova_metadata
740d8d5b3960  registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.1                    kolla_start           2 hours ago    Up 4 minutes ago          nova_api
92efa265b0ee  registry.redhat.io/rhosp-rhel8/openstack-glance-api:16.1                  kolla_start           2 hours ago    Up 4 minutes ago          glance_api
e526733ed6de  registry.redhat.io/rhosp-rhel8/openstack-placement-api:16.1               kolla_start           2 hours ago    Up 4 minutes ago          placement_api
099ad2995384  registry.redhat.io/rhosp-rhel8/openstack-neutron-metadata-agent-ovn:16.1  kolla_start           2 hours ago    Up 7 minutes ago          ovn_metadata_agent
86cfafece23d  registry.redhat.io/rhosp-rhel8/openstack-ovn-controller:16.1              kolla_start           2 hours ago    Up 4 minutes ago          ovn_controller
473a9e76e751  registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1                kolla_start           2 hours ago    Up 4 minutes ago          swift_rsync
7e6c057c23bb  registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1                kolla_start           2 hours ago    Up 3 minutes ago          swift_object_updater
bf9b6aa31e60  registry.redhat.io/rhosp-rhel8/openstack-swift-object:16.1                kolla_start           2 hours ago    Up 4 minutes ago          swift_object_server
542fa4615d5e  registry.redhat.io/rhosp-rhel8/openstack-swift-proxy-server:16.1          kolla_start           2 hours ago    Up 3 minutes ago          swift_object_expirer
4b51eed525e3  registry.redhat.io/rhosp-rhel8/openstack-swift-container:16.1             kolla_start           2 hours ago    Up 4 minutes ago          swift_container_updater
77bced83e17a  registry.redhat.io/rhosp-rhel8/openstack-swift-container:16.1             kolla_start           2 hours ago    Up 3 minutes ago          swift_container_server
aa75a0ec8689  registry.redhat.io/rhosp-rhel8/openstack-swift-account:16.1               kolla_start           2 hours ago    Up 3 minutes ago          swift_account_server
dd0b34104073  registry.redhat.io/rhosp-rhel8/openstack-swift-account:16.1               kolla_start           2 hours ago    Up 4 minutes ago          swift_account_reaper
68245e7185eb  registry.redhat.io/rhosp-rhel8/openstack-nova-novncproxy:16.1             kolla_start           2 hours ago    Up 4 minutes ago          nova_vnc_proxy
629347d0a808  registry.redhat.io/rhosp-rhel8/openstack-nova-scheduler:16.1              kolla_start           2 hours ago    Up 42 seconds ago         nova_scheduler
8f12ca348c92  registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.1                kolla_start           2 hours ago    Up 4 minutes ago          nova_migration_target
d76cb47095e2  registry.redhat.io/rhosp-rhel8/openstack-nova-conductor:16.1              kolla_start           2 hours ago    Up 35 seconds ago         nova_conductor
cc75b9d18720  registry.redhat.io/rhosp-rhel8/openstack-neutron-server-ovn:16.1          kolla_start           2 hours ago    Up 4 minutes ago          neutron_api
07b27271752f  registry.redhat.io/rhosp-rhel8/openstack-cron:16.1                        kolla_start           2 hours ago    Up 4 minutes ago          logrotate_crond
eeb687f1ee3e  registry.redhat.io/rhosp-rhel8/openstack-cinder-scheduler:16.1            kolla_start           2 hours ago    Up 4 minutes ago          cinder_scheduler
445a8e8666f9  registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.1                  kolla_start           2 hours ago    Up 4 minutes ago          cinder_api_cron
502049a4d16d  registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.1                  kolla_start           2 hours ago    Up 7 minutes ago          cinder_api
2cfe9182bef3  registry.redhat.io/rhosp-rhel8/openstack-keystone:16.1                    kolla_start           3 hours ago    Up 4 minutes ago          keystone
f7a2f0a07a9e  registry.redhat.io/rhosp-rhel8/openstack-iscsid:16.1                      kolla_start           3 hours ago    Up 4 minutes ago          iscsid
209ff53e823d  registry.redhat.io/rhosp-rhel8/openstack-nova-libvirt:16.1                kolla_start           3 hours ago    Up 3 minutes ago          nova_libvirt
7dbc2dc463d2  registry.redhat.io/rhosp-rhel8/openstack-nova-libvirt:16.1                kolla_start           4 hours ago    Up 4 minutes ago          nova_virtlogd
d9b6ed6f87ac  registry.redhat.io/rhosp-rhel8/openstack-horizon:16.1                     kolla_start           4 hours ago    Up 4 minutes ago          horizon
6bfbc1b9c5e9  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.1                     kolla_start           4 hours ago    Up 4 minutes ago          clustercheck
fcf7b19a54d3  registry.redhat.io/rhosp-rhel8/openstack-memcached:16.1                   kolla_start           4 hours ago    Up 4 minutes ago          memcached
===

I still have the same problem. Any other suggestion?

Comment 3 Luca Miccini 2021-05-18 15:40:59 UTC
I think one thing you definitely want to fix is to have --control-virtual-ip and --local-ip using different IPs, for example 192.168.24.3 and 192.168.24.2 respectively:

sudo openstack tripleo deploy --templates --control-virtual-ip=$IP --local-ip=$IP/$NETMASK \
-e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
-e $HOME/containers-prepare-parameters.yaml \
-e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone

if that fails maybe I can have a look at your env to figure this out?

Comment 4 Alexon Oliveira 2021-05-19 18:52:30 UTC
(In reply to Luca Miccini from comment #3)
> I think one thing you definitely want to fix is to have --control-virtual-ip
> and --local-ip using different IPs, for example 192.168.24.3 and
> 192.168.24.2 respectively:
> 
> sudo openstack tripleo deploy --templates --control-virtual-ip=$IP
> --local-ip=$IP/$NETMASK \
> -e
> /usr/share/openstack-tripleo-heat-templates/environments/standalone/
> standalone-tripleo.yaml \
> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
> -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
> -e $HOME/containers-prepare-parameters.yaml \
> -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone
> 
> if that fails maybe I can have a look at your env to figure this out?

Luca,

Different configs, same results. Sure, you can take a look at my configs. What do you want to check? I can attach here the config files you'll need.

Comment 5 Luca Miccini 2021-05-20 07:51:14 UTC
(In reply to Alexon Oliveira from comment #4)
> (In reply to Luca Miccini from comment #3)
> > I think one thing you definitely want to fix is to have --control-virtual-ip
> > and --local-ip using different IPs, for example 192.168.24.3 and
> > 192.168.24.2 respectively:
> > 
> > sudo openstack tripleo deploy --templates --control-virtual-ip=$IP
> > --local-ip=$IP/$NETMASK \
> > -e
> > /usr/share/openstack-tripleo-heat-templates/environments/standalone/
> > standalone-tripleo.yaml \
> > -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
> > -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
> > -e $HOME/containers-prepare-parameters.yaml \
> > -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone
> > 
> > if that fails maybe I can have a look at your env to figure this out?
> 
> Luca,
> 
> Different configs, same results. Sure, you can take a look at my configs.
> What do you want to check? I can attach here the config files you'll need.

can you attach or upload a sosreport somewhere? (btw I just realized that the command that I copy/pasted has the same variables for both the VIPs, I hope you used different IPs).

Cheers,
Luca

Comment 9 Luca Miccini 2021-05-26 18:06:09 UTC
Full List of Resources:
  * Container bundle: galera-bundle [cluster.common.tag/openstack-mariadb:pcmklatest]:
      * galera  (ocf::heartbeat:galera):        Stopped
      * galera-bundle-podman-0  (ocf::heartbeat:podman):        Stopped
      * galera-bundle-0 (ocf::pacemaker:remote):        Stopped
  * Container bundle: rabbitmq-bundle [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
      * rabbitmq        (ocf::heartbeat:rabbitmq-cluster):      Started rabbitmq-bundle-0
      * rabbitmq-bundle-podman-0        (ocf::heartbeat:podman):        Started openstack16
      * rabbitmq-bundle-0       (ocf::pacemaker:remote):        Started openstack16
  * Container bundle: redis-bundle [cluster.common.tag/openstack-redis:pcmklatest]:
      * redis   (ocf::heartbeat:redis): Master redis-bundle-0
      * redis-bundle-podman-0   (ocf::heartbeat:podman):        Started openstack16
      * redis-bundle-0  (ocf::pacemaker:remote):        Started openstack16
  * ip-192.168.3.2      (ocf::heartbeat:IPaddr2):       Stopped
  * Container bundle: haproxy-bundle [cluster.common.tag/openstack-haproxy:pcmklatest]:
      * haproxy-bundle-podman-0 (ocf::heartbeat:podman):        Stopped
  * Container bundle: ovn-dbs-bundle [cluster.common.tag/openstack-ovn-northd:pcmklatest]:
      * ovndb_servers   (ocf::ovn:ovndb-servers):       Master ovn-dbs-bundle-0
      * ovn-dbs-bundle-podman-0 (ocf::heartbeat:podman):        Started openstack16
      * ovn-dbs-bundle-0        (ocf::pacemaker:remote):        Started openstack16
  * Container bundle: openstack-cinder-volume [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
      * openstack-cinder-volume-podman-0        (ocf::heartbeat:podman):        Started openstack16


you haven't followed my advice to use a different Ip address for --control-virtual-ip :)

please try with 

sudo openstack tripleo deploy --templates --control-virtual-ip=192.168.3.3 --local-ip=192.168.3.2/24 \
-e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
-e $HOME/containers-prepare-parameters.yaml \
-e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone

Comment 10 Alexon Oliveira 2021-05-27 22:11:22 UTC
(In reply to Luca Miccini from comment #9)
> Full List of Resources:
>   * Container bundle: galera-bundle
> [cluster.common.tag/openstack-mariadb:pcmklatest]:
>       * galera  (ocf::heartbeat:galera):        Stopped
>       * galera-bundle-podman-0  (ocf::heartbeat:podman):        Stopped
>       * galera-bundle-0 (ocf::pacemaker:remote):        Stopped
>   * Container bundle: rabbitmq-bundle
> [cluster.common.tag/openstack-rabbitmq:pcmklatest]:
>       * rabbitmq        (ocf::heartbeat:rabbitmq-cluster):      Started
> rabbitmq-bundle-0
>       * rabbitmq-bundle-podman-0        (ocf::heartbeat:podman):       
> Started openstack16
>       * rabbitmq-bundle-0       (ocf::pacemaker:remote):        Started
> openstack16
>   * Container bundle: redis-bundle
> [cluster.common.tag/openstack-redis:pcmklatest]:
>       * redis   (ocf::heartbeat:redis): Master redis-bundle-0
>       * redis-bundle-podman-0   (ocf::heartbeat:podman):        Started
> openstack16
>       * redis-bundle-0  (ocf::pacemaker:remote):        Started openstack16
>   * ip-192.168.3.2      (ocf::heartbeat:IPaddr2):       Stopped
>   * Container bundle: haproxy-bundle
> [cluster.common.tag/openstack-haproxy:pcmklatest]:
>       * haproxy-bundle-podman-0 (ocf::heartbeat:podman):        Stopped
>   * Container bundle: ovn-dbs-bundle
> [cluster.common.tag/openstack-ovn-northd:pcmklatest]:
>       * ovndb_servers   (ocf::ovn:ovndb-servers):       Master
> ovn-dbs-bundle-0
>       * ovn-dbs-bundle-podman-0 (ocf::heartbeat:podman):        Started
> openstack16
>       * ovn-dbs-bundle-0        (ocf::pacemaker:remote):        Started
> openstack16
>   * Container bundle: openstack-cinder-volume
> [cluster.common.tag/openstack-cinder-volume:pcmklatest]:
>       * openstack-cinder-volume-podman-0        (ocf::heartbeat:podman):    
> Started openstack16
> 
> 
> you haven't followed my advice to use a different Ip address for
> --control-virtual-ip :)
> 
> please try with 
> 
> sudo openstack tripleo deploy --templates --control-virtual-ip=192.168.3.3
> --local-ip=192.168.3.2/24 \
> -e
> /usr/share/openstack-tripleo-heat-templates/environments/standalone/
> standalone-tripleo.yaml \
> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
> -r /usr/share/openstack-tripleo-heat-templates/roles/Standalone.yaml \
> -e $HOME/containers-prepare-parameters.yaml \
> -e $HOME/standalone_parameters.yaml --output-dir $HOME --standalone

I've done it. Didn't work either. Check the new logs I provided to you in the same place, please. Also, I did a fresh install with these recommendations. It didn't work.