Bug 1789063

Description of problem: Using RHOS_TRUNK-16.0-RHEL-8-20200107.n.5
deployment failed on Composable role (3ctl,3dbs,2net,2cpt,3msg) during
wait for container to start step2.

The error log are *very* hard to read ... so I may be missing
something here.

Using:

grep -Eo '^(TASK|fatal).*$' undercloud-0/home/stack/overcloud_install.log | sed -e 's/\\n/\
/g'  | grep -v "^b''" | less

I've found two plausible errors:

fatal: [database-0]: FAILED! => {"ansible_job_id": "921965978310.19454", "attempts": 578, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2", "rc": 1, "stderr": "$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-
mariadb:20200107.2
$ podman ps -a --filter label=managed_by=tripleo-Database --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=paunch --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=tripleo-Database --format {{.Names}} {{.Labels.container_name}}
b'mysql_bootstrap mysql_bootstrap\
mysql_data_ownership mysql_data_ownership\
'
$ podman ps -a --filter label=managed_by=tripleo-Database --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=paunch --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
Running container: mysql_restart_bundle
$ podman ps -a --filter label=container_name=mysql_restart_bundle --filter label=config_id=tripleo_step2 --format {{.Names}}
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
$ podman ps -a --filter label=container_name=mysql_restart_bundle --format {{.Names}}
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--format', '{{.Names}}']\"
Start container mysql_restart_bundle as mysql_restart_bundle.
$ podman run --name mysql_restart_bundle --label config_id=tripleo_step2 --label container_name=mysql_restart_bundle --label managed_by=tripleo-Database --label config_data={\"command\": \"/pacemaker_restart_bundle.sh galera-bundle mysql\", \"config_volume\": \"mysql\", \"detach\": f
alse, \"environment\": {\"TRIPLEO_MINOR_UPDATE\": \"\", \"TRIPLEO_CONFIG_HASH\": \"a5214c7a1f94d18a13a4107e268892b2\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:20200107.2\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 0, \"use
r\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/c
erts/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro\",
 \"/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro\", \"/dev/shm:/dev/shm:rw\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro\"]} --conmon-pidfile=/var/run/mysql_restart_bundle.pid --log-driver k8s-file --
log-opt path=/var/log/containers/stdouts/mysql_restart_bundle.log --env=TRIPLEO_CONFIG_HASH=a5214c7a1f94d18a13a4107e268892b2 --env=TRIPLEO_MINOR_UPDATE --net=host --ipc=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-t
rust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust
.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/var/lib/container-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro --volume=/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro --volume=/dev/shm:/dev
/shm:rw --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro --cpuset-cpus=0,1,2,3 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-mariadb:20200107.2 /pacemaker_restart_bundle.sh galera-bundle m
ysql
b\"Warning: This command is deprecated and will be removed. Please use 'pcs resource config' instead.\
\"
b\"Error: unable to find resource 'galera-bundle'\

and

fatal: [messaging-0]: FAILED! => {"ansible_job_id": "65073464675.18645", "attempts": 691, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2", "rc": 6, "stderr": "$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:20200107.2
$ podman ps -a --filter label=managed_by=tripleo-Messaging --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=paunch --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=tripleo-Messaging --format {{.Names}} {{.Labels.container_name}}
b'rabbitmq_bootstrap rabbitmq_bootstrap\
'
$ podman ps -a --filter label=managed_by=tripleo-Messaging --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
$ podman ps -a --filter label=managed_by=paunch --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}
Running container: rabbitmq_restart_bundle
$ podman ps -a --filter label=container_name=rabbitmq_restart_bundle --filter label=config_id=tripleo_step2 --format {{.Names}}
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id
$ podman ps -a --filter label=container_name=rabbitmq_restart_bundle --format {{.Names}}
Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--format', '{{.Names}}']\"
Start container rabbitmq_restart_bundle as rabbitmq_restart_bundle.
$ podman run --name rabbitmq_restart_bundle --label config_id=tripleo_step2 --label container_name=rabbitmq_restart_bundle --label managed_by=tripleo-Messaging --label config_data={\"command\": \"/pacemaker_restart_bundle.sh rabbitmq-bundle oslo_messaging_rpc\", \"config_volume\": \"rabbitmq\", \"detach\": false, \"environment\": {\"TRIPLEO_MINOR_UPDATE\": \"\", \"TRIPLEO_CONFIG_HASH\": \"cc76613a25c3f241487a2ae41e6db537\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:20200107.2\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 0, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro\", \"/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro\", \"/dev/shm:/dev/shm:rw\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/puppet-generated/rabbitmq/:/var/lib/kolla/config_files/src:ro\"]} --conmon-pidfile=/var/run/rabbitmq_restart_bundle.pid --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/rabbitmq_restart_bundle.log --env=TRIPLEO_CONFIG_HASH=cc76613a25c3f241487a2ae41e6db537 --env=TRIPLEO_MINOR_UPDATE --net=host --ipc=host --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/var/lib/container-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro --volume=/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro --volume=/dev/shm:/dev/shm:rw --volume=/etc/puppet:/etc/puppet:ro --volume=/var/lib/config-data/puppet-generated/rabbitmq/:/var/lib/kolla/config_files/src:ro --cpuset-cpus=0,1,2,3 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-rabbitmq:20200107.2 /pacemaker_restart_bundle.sh rabbitmq-bundle oslo_messaging_rpc
b\"Warning: This command is deprecated and will be removed. Please use 'pcs resource config' instead.\
\"
b\"Error: unable to find resource 'rabbitmq-bundle'\

How reproducible: Happened once in that job
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-16-from-passed_phase1-composable-ipv6-scale-up/33/