Bug 1871241

Summary: FFU 13z12 16.1.1 faling starting podman containers (baremetal deployment)
Product: Red Hat OpenStack Reporter: Jason Grosso <jgrosso>
Component: openstack-tripleoAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Arik Chernetsky <achernet>
Severity: unspecified Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: aschultz, jfrancoa, mburns
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-24 09:16:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
full_cmd_iine_output none

Description Jason Grosso 2020-08-21 17:41:04 UTC
Created attachment 1712206 [details]
full_cmd_iine_output

Description of problem:
TASK [Wait for containers to start for step 3 using paunch]

Version-Release number of selected component (if applicable):
FFU 13z12 16.1.1(latest passed phase2 

How reproducible:
I am trying to perform an FFU upgrade using https://gitlab.cee.redhat.com/osp16/ffwd2#run-the-undercloud-upgrade





Actual results:

Please see attachment for full log

TASK [Start containers for step 3 using paunch] *******************************************************************************************************************************************************************
Friday 21 August 2020  20:10:58 +0300 (0:00:00.105)       0:05:13.161 *********
changed: [sealusa40]

TASK [Wait for containers to start for step 3 using paunch] *******************************************************************************************************************************************************
Friday 21 August 2020  20:10:59 +0300 (0:00:00.364)       0:05:13.526 *********
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1200 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1199 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1198 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1197 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1196 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1195 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1194 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1193 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1192 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1191 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1190 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1189 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1188 retries left).
WAITING FOR COMPLETION: Wait for containers to start for step 3 using paunch (1187 retries left).
fatal: [sealusa40]: FAILED! => {"ansible_job_id": "88444727671.805453", "attempts": 15, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step3", "rc": 1, "stderr": "Error executing ['podman', 'container', 'exists', 'glance_api_db_sync_stein']: returned 1\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync_stein', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_api_db_sync_stein', '--format', '{{.Names}}']\"\nError executing ['podman', 'run', '--name', 'glance_api_db_sync_stein', '--label', 'config_id=tripleo_step3', '--label', 'container_name=glance_api_db_sync_stein', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": [\"/usr/bin/bootstrap_host_exec\", \"glance_api\", \"/container-config-scripts/glance_ffu_db_sync.sh\"], \"detach\": false, \"environment\": {\"KOLLA_BOOTSTRAP\": true, \"KOLLA_CONFIG_STRATEGY\": \"COPY_ALWAYS\", \"TRIPLEO_DEPLOY_IDENTIFIER\": \"1598029257\", \"TRIPLEO_CONFIG_HASH\": \"a6bd1aa8a1c299091fb1c9ce9cb82f31\"}, \"image\": \"sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp15-openstack-glance-api:20200115.1\", \"net\": \"host\", \"privileged\": false, \"start_order\": 0, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/glance:/var/log/glance:z\", \"/var/log/containers/httpd/glance:/var/log/httpd:z\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/glance_api:/var/lib/kolla/config_files/src:ro\", \"/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\", \"/var/lib/glance:/var/lib/glance:slave\", \"/var/lib/container-config-scripts/:/container-config-scripts/:ro\"]}', '--conmon-pidfile=/var/run/glance_api_db_sync_stein.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/glance_api_db_sync_stein.log', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=a6bd1aa8a1c299091fb1c9ce9cb82f31', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1598029257', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/glance:/var/log/glance:z', '--volume=/var/log/containers/httpd/glance:/var/log/httpd:z', '--volume=/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json', '--volume=/var/lib/config-data/puppet-generated/glance_api:/var/lib/kolla/config_files/src:ro', '--volume=/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro', '--volume=/var/lib/glance:/var/lib/glance:slave', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/:ro', 'sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp15-openstack-glance-api:20200115.1', '/usr/bin/bootstrap_host_exec', 'glance_api', '/container-config-scripts/glance_ffu_db_sync.sh']: returned 1\nError running ['podman', 'run', '--name', 'glance_api_db_sync_stein', '--label', 'config_id=tripleo_step3', '--label', 'container_name=glance_api_db_sync_stein', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": [\"/usr/bin/bootstrap_host_exec\", \"glance_api\", \"/container-config-scripts/glance_ffu_db_sync.sh\"], \"detach\": false, \"environment\": {\"KOLLA_BOOTSTRAP\": true, \"KOLLA_CONFIG_STRATEGY\": \"COPY_ALWAYS\", \"TRIPLEO_DEPLOY_IDENTIFIER\": \"1598029257\", \"TRIPLEO_CONFIG_HASH\": \"a6bd1aa8a1c299091fb1c9ce9cb82f31\"}, \"image\": \"sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp15-openstack-glance-api:20200115.1\", \"net\": \"host\", \"privileged\": false, \"start_order\": 0, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/glance:/var/log/glance:z\", \"/var/log/containers/httpd/glance:/var/log/httpd:z\", \"/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/glance_api:/var/lib/kolla/config_files/src:ro\", \"/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro\", \"/var/lib/glance:/var/lib/glance:slave\", \"/var/lib/container-config-scripts/:/container-config-scripts/:ro\"]}', '--conmon-pidfile=/var/run/glance_api_db_sync_stein.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/glance_api_db_sync_stein.log', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=a6bd1aa8a1c299091fb1c9ce9cb82f31', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1598029257', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/glance:/var/log/glance:z', '--volume=/var/log/containers/httpd/glance:/var/log/httpd:z', '--volume=/var/lib/kolla/config_files/glance_api.json:/var/lib/kolla/config_files/config.json', '--volume=/var/lib/config-data/puppet-generated/glance_api:/var/lib/kolla/config_files/src:ro', '--volume=/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro', '--volume=/var/lib/glance:/var/lib/glance:slave', '--volume=/var/lib/container-config-scripts/:/container-config-scripts/:ro', 'sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp15-openstack-glance-api:20200115.1', '/usr/bin/bootstrap_host_exec', 'glance_api', '/container-config-scripts/glance_ffu_db_sync.sh']. [1]\n\nstdout: \nstderr: Cannot find config file: /etc/puppet/hiera.yaml\n\nError executing ['podman', 'container', 'exists', 'heat_engine_db_sync']: returned 1\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync', '--format', '{{.Names}}']\"\nError executing ['podman', 'run', '--name', 'heat_engine_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=heat_engine_db_sync', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec heat_engine su heat -s /bin/bash -c \\'heat-manage db_sync\\'\", \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1598029257\", \"TRIPLEO_CONFIG_HASH\": \"a57c781e7258833d6af4bc0b6f9e05a3-a57c781e7258833d6af4bc0b6f9e05a3\"}, \"image\": \"sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20200730.1\", \"net\": \"host\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/heat:/var/log/heat:z\", \"/var/lib/config-data/heat/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/heat/etc/heat/:/etc/heat/:ro\"]}', '--conmon-pidfile=/var/run/heat_engine_db_sync.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/heat_engine_db_sync.log', '--env=TRIPLEO_CONFIG_HASH=a57c781e7258833d6af4bc0b6f9e05a3-a57c781e7258833d6af4bc0b6f9e05a3', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1598029257', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/heat:/var/log/heat:z', '--volume=/var/lib/config-data/heat/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro', '--volume=/var/lib/config-data/heat/etc/heat/:/etc/heat/:ro', 'sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20200730.1', '/usr/bin/bootstrap_host_exec', 'heat_engine', 'su', 'heat', '-s', '/bin/bash', '-c', \"'heat-manage\", \"db_sync'\"]: returned 1\nError running ['podman', 'run', '--name', 'heat_engine_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=heat_engine_db_sync', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec heat_engine su heat -s /bin/bash -c \\'heat-manage db_sync\\'\", \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1598029257\", \"TRIPLEO_CONFIG_HASH\": \"a57c781e7258833d6af4bc0b6f9e05a3-a57c781e7258833d6af4bc0b6f9e05a3\"}, \"image\": \"sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20200730.1\", \"net\": \"host\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/heat:/var/log/heat:z\", \"/var/lib/config-data/heat/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/heat/etc/heat/:/etc/heat/:ro\"]}', '--conmon-pidfile=/var/run/heat_engine_db_sync.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/heat_engine_db_sync.log', '--env=TRIPLEO_CONFIG_HASH=a57c781e7258833d6af4bc0b6f9e05a3-a57c781e7258833d6af4bc0b6f9e05a3', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1598029257', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/log/containers/heat:/var/log/heat:z', '--volume=/var/lib/config-data/heat/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro', '--volume=/var/lib/config-data/heat/etc/heat/:/etc/heat/:ro', 'sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-heat-engine:16.1_20200730.1', '/usr/bin/bootstrap_host_exec', 'heat_engine', 'su', 'heat', '-s', '/bin/bash', '-c', \"'heat-manage\", \"db_sync'\"]. [1]\n\nstdout: \nstderr: Cannot find config file: /etc/puppet/hiera.yaml\n\nError executing ['podman', 'container', 'exists', 'heat_engine_db_sync_stein']: returned 1\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync_stein', '--filter', 'label=config_id=tripleo_step3', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=heat_engine_db_sync_stein', '--format', '{{.Names}}']\"\nError executing ['podman', 'run', '--name', 'heat_engine_db_sync_stein', '--label', 'config_id=tripleo_step3', '--label', 'container_name=heat_engine_db_sync_stein', '--label', 'managed_by=tripleo-Undercloud', '--label', 'config_data={\"command\": \"/usr/bin/bootstrap_host_exec heat_engine su heat -s /bin/bash -c \\'heat-manage db_sync\\'\", \"detach\": false, \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1598029257\", \"TRIPLEO_CONFIG_HASH\": \"a57c781e7258833d6af4bc0b6f9e05a3-a57c781e7258833d6af4bc0b6f9e05a3\"}, \"image\": \"sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp15-openstack-heat-engine:20200115.1\", \"net\": \"host\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/puppet:/etc/puppet:ro\", \"/var/log/containers/heat:/var/log/heat:z\", \"/var/lib/config-data/heat/etc/my.cnf.d/tripleo.cnf:/etc/my.cnf.d/tripleo.cnf:ro\", \"/var/lib/config-data/heat/etc/heat/:/etc/heat/:ro\"]}', '--conmon-pidfile=/var/run/heat_engine_db_sync_stein.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/heat_engine_db_sync_stein.log', 


I kicked of the undercloud upgrade last night and it failed, I tried it again this afternoon 

(undercloud) [stack@sealusa40 ~]$ sudo podman ps 
CONTAINER ID  IMAGE                                                                                    COMMAND      CREATED       STATUS           PORTS  NAMES
951153cbeb0e  sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20200730.1   kolla_start  22 hours ago  Up 22 hours ago         keystone
5aaaed492d0e  sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200730.1     kolla_start  22 hours ago  Up 22 hours ago         iscsid
2e0b811045c0  sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-mariadb:16.1_20200730.1    kolla_start  22 hours ago  Up 22 hours ago         mysql
32a6d62d9f0a  sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-rabbitmq:16.1_20200730.1   kolla_start  22 hours ago  Up 22 hours ago         rabbitmq
10a80b433236  sealusa40.ctlplane.localdomain:8787/rh-osbs/rhosp16-openstack-memcached:16.1_20200730.1  kolla_start  22 hours ago  Up 22 hours ago         memcached


I also noticed starting at 20:09:15 +0300

TASK [Debug output for task: Run puppet host configuration for step 1] ********************************************************************************************************************************************
Friday 21 August 2020  20:09:15 +0300 (0:00:06.636)       0:03:29.619 *********
changed: [sealusa40] => {
    "failed_when_result": false,
    "puppet_host_outputs.stdout_lines | default([]) | union(puppet_host_outputs.stderr_lines | default([]))": [
        "<13>Aug 21 20:09:08 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/
Expected results:

undercloud successfully upgrades


Additional info:
baremetal system
sealusa40.mobius.lab.eng.rdu2.redhat.com   stack\12345678

Comment 1 Jose Luis Franco 2020-08-24 09:16:21 UTC
Closing this BZ as it is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1829609. The workaround to solve this problem is to reinstall containers-selinux RPM and the cause is that the right containers-tools DNF module wasn't enabled before starting the Undercloud Upgrade, as a consequence a bad version of containers-selinux and podman got installed and caused the failure.

This should be solved by the following BZ in the future, in the case customers miss to set up the right DNF module we will ensure it is enabled in tripleo-heat-templates: https://bugzilla.redhat.com/show_bug.cgi?id=1866479

*** This bug has been marked as a duplicate of bug 1829609 ***