UC install fails with AttributeError: 'PodmanRunner' object has no attribute 'container_exists' Description of problem: > openstack undercloud install fails, it's ansible output indicates that it failed during waiting for containers to start for step3 using paunch ... although these containers seems exist and already exited (later after the failure). (At least some I've checked like heat_engine_db_sync, neutron_ovs_bridge, ... missing seems mysql_sync_credentials though it was checking for one named mysql instead and that one is there.) But interesting is that the waiting for containers loop/retries seems aborted/failed too early, before all the printed-out retries amount was reached. There seems to be two outstanding interesting Tracebacks in the output too. First on stdout, maybe it could be indication for the second traceback: > ValueError: No start of json char found" though i have no clue what json could be missing where. Second, and seems possibly important for the failure is: > AttributeError: 'PodmanRunner' object has no attribute 'container_exists' as that one maybe is related to why it stops RETRYING THE WAIT before the retries/attempts count reaches the limit. (Notice the 'attemps' and 'retries left' counts in the output.) Final failing task in output from undercloud installation: > 2021-04-09 22:22:40.618 6321 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1183 retries left). > 2021-04-09 22:22:43.812 6321 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] fatal: [undercloud-0]: FAILED! => > {"ansible_job_id": "160701373548.32264", > "attempts": 19, > "changed": false, > "cmd": "/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py", > "data": "", > "finished": 1, > "msg": " > Traceback (most recent call last): > File \"/tmp/ansible_async_wrapper_payload_fnqz7765/ansible_async_wrapper_payload.zip/ansible/modules/utilities/logic/async_wrapper.py\", line 166, in _run_module > File \"/tmp/ansible_async_wrapper_payload_fnqz7765/ansible_async_wrapper_payload.zip/ansible/modules/utilities/logic/async_wrapper.py\", line 94, in _filter_non_json_lines > ValueError: No start of json char found", > "stderr": " > Error executing ['podman', 'container', 'exists', 'heat_engine_db_sync']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_log']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'neutron_ovs_bridge']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'nova_statedir_owner']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'swift_copy_rings']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'zaqar_db_sync']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'glance_api_db_sync']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_dnsmasq_dhcp_hostsdir']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'keystone_db_sync']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'mistral_db_sync']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--format', '{{.Names}}']\" > Error executing ['podman', 'container', 'exists', 'mysql_sync_credentials']: returned 1 > Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id > Traceback (most recent call last): > File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 102, in <module> > _ansiballz_main() > File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 94, in _ansiballz_main > invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS) > File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 40, in invoke_module > runpy.run_module(mod_name='ansible.modules.paunch', init_globals=None, run_name='__main__', alter_sys=True) > File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module > return _run_module_code(code, init_globals, run_name, mod_spec) > File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code > mod_name, mod_spec, pkg_name, script_name) > File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code > exec(code, run_globals) > File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 253, in <module> > File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 249, in main > File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 180, in __init__ > File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 204, in paunch_apply > File \"/usr/lib/python3.6/site-packages/paunch/__init__.py\", line 84, in apply > return builder.apply() > File \"/usr/lib/python3.6/site-packages/paunch/builder/base.py\", line 132, in apply > if not c_name or not self.runner.container_running(c_name): > File \"/usr/lib/python3.6/site-packages/paunch/runner.py\", line 498, in container_running > if not self.container_exists(container): > AttributeError: 'PodmanRunner' object has no attribute 'container_exists' > ", > "stderr_lines": ["Error executing ['podman', 'container', 'exists', 'heat_engine_db_sync']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_log']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'neutron_ovs_bridge']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'nova_statedir_owner']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'swift_copy_rings']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'zaqar_db_sync']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'glance_api_db_sync']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_dnsmasq_dhcp_hostsdir']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'keystone_db_sync']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'mistral_db_sync']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--format', '{{.Names}}']\"", "Error executing ['podman', 'container', 'exists', 'mysql_sync_credentials']: returned 1", "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", "Traceback (most recent call last):", " File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 102, in <module>", " _ansiballz_main()", " File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 94, in _ansiballz_main", " invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)", " File \"/root/.ansible/tmp/ansible-tmp-1618021305.0634766-32252-67876202080949/AnsiballZ_paunch.py\", line 40, in invoke_module", " runpy.run_module(mod_name='ansible.modules.paunch', init_globals=None, run_name='__main__', alter_sys=True)", " File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module", " return _run_module_code(code, init_globals, run_name, mod_spec)", " File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code", " mod_name, mod_spec, pkg_name, script_name)", " File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", " exec(code, run_globals)", " File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 253, in <module>", " File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 249, in main", " File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 180, in __init__", " File \"/tmp/ansible_paunch_payload_1o7ncm9z/ansible_paunch_payload.zip/ansible/modules/paunch.py\", line 204, in paunch_apply", " File \"/usr/lib/python3.6/site-packages/paunch/__init__.py\", line 84, in apply", " return builder.apply()", " File \"/usr/lib/python3.6/site-packages/paunch/builder/base.py\", line 132, in apply", " if not c_name or not self.runner.container_running(c_name):", " File \"/usr/lib/python3.6/site-packages/paunch/runner.py\", line 498, in container_running", " if not self.container_exists(container):", "AttributeError: 'PodmanRunner' object has no attribute 'container_exists'"] > } > 2021-04-09 22:22:43.814 6321 WARNING tripleoclient.v1.tripleo_deploy.Deploy List of containers in podman as gathered in post-deployment logs: > + podman ps --all > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES > e8f42e3a918f undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mistral-api:16.1_20210409.1 /usr/bin/bootstra... About a minute ago Exited (0) About a minute ago mistral_db_sync > 015917eee8a5 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20210409.1 /usr/bin/bootstra... About a minute ago Exited (0) About a minute ago keystone_db_sync > 613bcdb4cace undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20210409.1 /bin/bash -c chow... About a minute ago Exited (0) About a minute ago ironic_inspector_init_dnsmasq_dhcp_hostsdir > 1806b7646553 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-glance-api:16.1_20210409.1 /usr/bin/bootstra... About a minute ago Exited (0) About a minute ago glance_api_db_sync > 19b5c7591ef4 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-zaqar-wsgi:16.1_20210409.1 /usr/bin/bootstra... 2 minutes ago Exited (0) About a minute ago zaqar_db_sync > 11707a69528b undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-proxy-server:16.1_20210409.1 /bin/bash -c cp -... 2 minutes ago Exited (0) 2 minutes ago swift_copy_rings > d2cdefad4a7f undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute-ironic:16.1_20210409.1 /container-config... 2 minutes ago Exited (0) 2 minutes ago nova_statedir_owner > 2847534ac350 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-server:16.1_20210409.1 puppet apply --mo... 2 minutes ago Exited (0) 2 minutes ago neutron_ovs_bridge > b7518aa0981e undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20210409.1 /bin/bash -c chow... 2 minutes ago Exited (0) 2 minutes ago ironic_inspector_init_log > 9a5804495f30 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20210409.1 /usr/bin/bootstra... 2 minutes ago Exited (0) 2 minutes ago heat_engine_db_sync > 343e58bf779a undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20210409.1 /container-config... 3 minutes ago Exited (0) 3 minutes ago mysql_neutron_db_rename > 5a77b7444418 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20210409.1 kolla_start 3 minutes ago Up 3 minutes ago mysql > 7151839ba529 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20210409.1 /bin/bash -c chow... 3 minutes ago Exited (0) 3 minutes ago placement_init_log > b5342bb646ef undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20210409.1 bash -ec if [ -e ... 3 minutes ago Exited (0) 3 minutes ago mysql_bootstrap > 1642db162a2d undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20210409.1 /bin/bash -c chow... 3 minutes ago Exited (0) 3 minutes ago keystone_init_log > c3723baafa7d undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-l3-agent:16.1_20210409.1 /container_puppet... 3 minutes ago Exited (0) 3 minutes ago create_keepalived_wrapper > 9efffa4bd34b undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-dhcp-agent:16.1_20210409.1 /container_puppet... 4 minutes ago Exited (0) 3 minutes ago create_dnsmasq_wrapper > b8ad12e42644 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-zaqar-wsgi:16.1_20210409.1 /bin/bash -c chmo... 4 minutes ago Exited (0) 4 minutes ago zaqar_init_log > d40f6108408f undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-tempest:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago tempest_init_logs > 027395079eb3 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-conductor:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago nova_conductor_init_log > 2276bea63917 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago nova_api_init_logs > 10a24b23cf33 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-server:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago neutron_init_logs > 09aef42775c9 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mistral-api:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago mistral_init_logs > 25ac69f3d166 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-api:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago ironic_init_logs > 1583923cd4dc undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago heat_init_log > 5943be90a231 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-glance-api:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago glance_init_logs > e113ecdeeee1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:16.1_20210409.1 kolla_start 4 minutes ago Up 4 minutes ago rabbitmq > 7777cc7329f5 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:16.1_20210409.1 kolla_start 4 minutes ago Exited (0) 4 minutes ago rabbitmq_bootstrap > 8f2f98a270d9 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-haproxy:16.1_20210409.1 kolla_start 4 minutes ago Up 4 minutes ago haproxy > 05e4aa348249 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago rabbitmq_init_logs > 4a91e802c111 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20210409.1 /bin/bash -c chow... 4 minutes ago Exited (0) 4 minutes ago mysql_init_logs > 5bec963761eb undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-memcached:16.1_20210409.1 kolla_start 4 minutes ago Up 4 minutes ago memcached > 27e50e1eb55e undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-keepalived:16.1_20210409.1 /usr/local/bin/ko... 4 minutes ago Up 4 minutes ago keepalived Happens in OSP Phase1 CI run, with compose RHOS-16.1-RHEL-8-20210409.n.2. Ans it seems to happen repeatedly / can be reproduced.
Some more details: > python3-paunch.noarch 5.3.3-1.20210310104358.ed2c015.el8ost @rhelosp-16.1 was missing newer patchset of https://review.opendev.org/c/openstack/paunch/+/769186 in which the issue was already fixed. Verified that with newer version > python3-paunch.noarch 5.3.3-1.20210412123423.ed2c015.el8ost @rhelosp-16.1 deployment passed and shows no signs of the issue. Snippet of 'undercloud install' output when it worked: > ... > TASK [Wait for containers to start for step 3 using paunch] ******************** > Tuesday 13 April 2021 09:21:43 -0400 (0:00:00.366) 0:09:37.421 ********* > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1200 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1199 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1198 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1197 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1196 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1195 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1194 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1193 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1192 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1191 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1190 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1189 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1188 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1187 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1186 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1185 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1184 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1183 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1182 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1181 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1180 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1179 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1178 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1177 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1176 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1175 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1174 retries left). > WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1173 retries left). > ok: [undercloud-0] > > TASK [Debug output for task: Start containers for step 3] ********************** > Tuesday 13 April 2021 09:23:13 -0400 (0:01:29.512) 0:11:06.933 ********* > ok: [undercloud-0] => { > "failed_when_result": false, > "start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [ > "2021-04-13 13:21:47.484 14 INFO migrate.versioning.api [-] 70 -> 71... \u001b[00m", > "2021-04-13 13:21:47.622 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.622 14 INFO migrate.versioning.api [-] 71 -> 72... \u001b[00m", > "2021-04-13 13:21:47.646 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.646 14 INFO migrate.versioning.api [-] 72 -> 73... \u001b[00m", > "2021-04-13 13:21:47.684 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.684 14 INFO migrate.versioning.api [-] 73 -> 74... \u001b[00m", > "2021-04-13 13:21:47.689 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.689 14 INFO migrate.versioning.api [-] 74 -> 75... \u001b[00m", > "2021-04-13 13:21:47.694 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.694 14 INFO migrate.versioning.api [-] 75 -> 76... \u001b[00m", > "2021-04-13 13:21:47.699 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.699 14 INFO migrate.versioning.api [-] 76 -> 77... \u001b[00m", > "2021-04-13 13:21:47.704 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.705 14 INFO migrate.versioning.api [-] 77 -> 78... \u001b[00m", > "2021-04-13 13:21:47.709 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.709 14 INFO migrate.versioning.api [-] 78 -> 79... \u001b[00m", > "2021-04-13 13:21:47.763 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.763 14 INFO migrate.versioning.api [-] 79 -> 80... \u001b[00m", > "2021-04-13 13:21:47.799 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.799 14 INFO migrate.versioning.api [-] 80 -> 81... \u001b[00m", > "2021-04-13 13:21:47.803 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.804 14 INFO migrate.versioning.api [-] 81 -> 82... \u001b[00m", > "2021-04-13 13:21:47.808 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.808 14 INFO migrate.versioning.api [-] 82 -> 83... \u001b[00m", > "2021-04-13 13:21:47.813 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.814 14 INFO migrate.versioning.api [-] 83 -> 84... \u001b[00m", > "2021-04-13 13:21:47.818 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.819 14 INFO migrate.versioning.api [-] 84 -> 85... \u001b[00m", > "2021-04-13 13:21:47.824 14 INFO migrate.versioning.api [-] done\u001b[00m", > "2021-04-13 13:21:47.824 14 INFO migrate.versioning.api [-] 85 -> 86... \u001b[00m", > "2021-04-13 13:21:47.868 14 INFO migrate.versioning.api [-] done\u001b[00m", > "", > "b493bba585523d812f8af6f94972af93d64f5775970cadc7183d4502e6e10ab0", > "\u001b[0;32mInfo: Loading facts\u001b[0m", > "\u001b[mNotice: Compiled catalog for undercloud-0.redhat.local in environment production in 0.60 seconds\u001b[0m", > "\u001b[0;32mInfo: Applying configuration version '1618320121'\u001b[0m", > "\u001b[0;32mInfo: Creating state file /var/lib/puppet/state/state.yaml\u001b[0m", > "\u001b[mNotice: Applied catalog in 0.15 seconds\u001b[0m", > "INFO:nova_statedir:Applying nova statedir ownership", > "INFO:nova_statedir:Target ownership for /var/lib/nova: 42436:42436", > "INFO:nova_statedir:Checking uid: 0 gid: 0 path: /var/lib/nova/", > "INFO:nova_statedir:Changing ownership of /var/lib/nova from 0:0 to 42436:42436", > "INFO:nova_statedir:Setting selinux context of /var/lib/nova to system_u:object_r:container_file_t:s0", > "INFO:nova_statedir:Nova statedir ownership complete", > "'/swift_ringbuilder/etc/swift/account.ring.gz' -> '/etc/swift/account.ring.gz'", > "'/swift_ringbuilder/etc/swift/container.ring.gz' -> '/etc/swift/container.ring.gz'", > "'/swift_ringbuilder/etc/swift/object.ring.gz' -> '/etc/swift/object.ring.gz'", > "'/swift_ringbuilder/etc/swift/account.builder' -> '/etc/swift/account.builder'", > "'/swift_ringbuilder/etc/swift/container.builder' -> '/etc/swift/container.builder'", > "'/swift_ringbuilder/etc/swift/object.builder' -> '/etc/swift/object.builder'", > "'/swift_ringbuilder/etc/swift/backups' -> '/etc/swift/backups'", > "'/swift_ringbuilder/etc/swift/backups/1618319983.account.builder' -> '/etc/swift/backups/1618319983.account.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319983.object.builder' -> '/etc/swift/backups/1618319983.object.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319984.container.builder' -> '/etc/swift/backups/1618319984.container.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319986.object.builder' -> '/etc/swift/backups/1618319986.object.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319986.object.ring.gz' -> '/etc/swift/backups/1618319986.object.ring.gz'", > "'/swift_ringbuilder/etc/swift/backups/1618319987.account.builder' -> '/etc/swift/backups/1618319987.account.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319987.account.ring.gz' -> '/etc/swift/backups/1618319987.account.ring.gz'", > "'/swift_ringbuilder/etc/swift/backups/1618319987.container.builder' -> '/etc/swift/backups/1618319987.container.builder'", > "'/swift_ringbuilder/etc/swift/backups/1618319987.container.ring.gz' -> '/etc/swift/backups/1618319987.container.ring.gz'", > "Upgraded database to: train_expand01, current revision(s): train_expand01", > "Database migration is up to date. No migration needed.", > "Upgraded database to: train_contract01, current revision(s): train_contract01", > "Database is synced successfully.", > "3ec9aa268f497340d11b63d4cf57f72b66d15416bb571ff1300f3178a1a0039c", > "Running upgrade for neutron ...", > "OK", > "Running upgrade for networking-l2gw ...", > "Running upgrade for networking-sfc ...", > "Running upgrade for neutron-dynamic-routing ...", > "Running upgrade for vmware-nsx ...", > "cf3b1bee53908a6f49a35163178a2aaa5cef98f49d6b80eb766001adf41f66e9", > "87f503bcd3299a92881eb7e26fe026537f94d716940448aae7d6b554dfcb29f8", > "4072327a417adde903b0ea89f3d3ab5107a96c721e904682dd672546b86bf604", > "No data present in nova database - nothing to migrate (new deployment?)", > "(cellv2) Creating default cell_v2 cell", > "Error executing ['podman', 'container', 'exists', 'heat_engine_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_log']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_log', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'neutron_ovs_bridge']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_ovs_bridge', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'nova_statedir_owner']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_statedir_owner', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'swift_copy_rings']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_copy_rings', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'zaqar_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=zaqar_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'glance_api_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'ironic_inspector_init_dnsmasq_dhcp_hostsdir']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_init_dnsmasq_dhcp_hostsdir', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'keystone_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'mistral_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mistral_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'mysql_sync_credentials']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Error executing ['podman', 'container', 'exists', 'neutron_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'swift_setup_srv']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_setup_srv', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=swift_setup_srv', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'ironic_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'iscsid']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=iscsid', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=iscsid', '--format', '{{.Names}}']\"", > "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_iscsid.service → /etc/systemd/system/tripleo_iscsid.service.", > "Created symlink /etc/systemd/system/timers.target.wants/tripleo_iscsid_healthcheck.timer → /etc/systemd/system/tripleo_iscsid_healthcheck.timer.", > "Created symlink /etc/systemd/system/tripleo_iscsid.service.requires/tripleo_iscsid_healthcheck.timer → /etc/systemd/system/tripleo_iscsid_healthcheck.timer.", > "Error executing ['podman', 'container', 'exists', 'keystone']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone', '--format', '{{.Names}}']\"", > "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_keystone.service → /etc/systemd/system/tripleo_keystone.service.", > "Created symlink /etc/systemd/system/timers.target.wants/tripleo_keystone_healthcheck.timer → /etc/systemd/system/tripleo_keystone_healthcheck.timer.", > "Created symlink /etc/systemd/system/tripleo_keystone.service.requires/tripleo_keystone_healthcheck.timer → /etc/systemd/system/tripleo_keystone_healthcheck.timer.", > "Error executing ['podman', 'container', 'exists', 'ironic_inspector_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'ironic_inspector_get_ipa']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_get_ipa', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=ironic_inspector_get_ipa', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'keystone_bootstrap']: returned 1", > "Error executing ['podman', 'container', 'exists', 'nova_api_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'placement_api_db_extract_data_from_nova_api']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_api_db_extract_data_from_nova_api', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_api_db_extract_data_from_nova_api', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'nova_api_map_cell0']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_map_cell0', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_map_cell0', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'placement_api_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_api_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_api_db_sync', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'nova_api_ensure_default_cell']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_ensure_default_cell', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_ensure_default_cell', '--format', '{{.Names}}']\"", > "Error executing ['podman', 'container', 'exists', 'nova_db_sync']: returned 1", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id", > "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_db_sync', '--format', '{{.Names}}']\"" > ] > } > > TASK [Manage containers for step 3 with tripleo-ansible] *********************** > Tuesday 13 April 2021 09:23:13 -0400 (0:00:00.068) 0:11:07.001 ********* > skipping: [undercloud-0] > > TASK [Clean container_puppet_tasks for undercloud-0 step 3] ******************** > Tuesday 13 April 2021 09:23:13 -0400 (0:00:00.066) 0:11:07.067 ********* > ok: [undercloud-0] > ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.6 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2097