issue was a difference between pacemaker rpms in the containers vs on the host.
On the host:
pacemaker-remote-1.1.21-4.el7.x86_64
pacemaker-libs-1.1.21-4.el7.x86_64
pcs-0.9.168-4.el7.x86_64
libqb-1.0.1-9.el7.x86_64
corosync-2.4.5-4.el7.x86_64
Whereas in the container:
sudo podman run -it --net=host --rm --name=foo registry-proxy.engineering.redhat.
com/rh-osbs/rhosp13-openstack-haproxy:20200303.1 sh -c 'rpm -q pacemaker corosync libqb pcs'
pacemaker-1.1.20-5.el7_7.2.x86_64
corosync-2.4.3-6.el7_7.1.x86_64
libqb-1.0.1-7.el7.x86_64
pcs-0.9.167-3.el7_7.1.x86_64
Description of problem: core_puddle: 2018-06-21.2 Overcloud deploy fails: 2020-04-04 11:57:40Z [overcloud.AllNodesDeploySteps.NetworkerDeployment_Step2.0]: CREATE_COMPLETE state changed 2020-04-04 11:57:58Z [overcloud.AllNodesDeploySteps.NetworkerDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment a578cb8a-7303-49bb-8807-1fa65575fbe9 succeeded 2020-04-04 11:57:59Z [overcloud.AllNodesDeploySteps.NetworkerDeployment_Step2.1]: CREATE_COMPLETE state changed 2020-04-04 11:57:59Z [overcloud.AllNodesDeploySteps.NetworkerDeployment_Step2]: CREATE_COMPLETE Stack CREATE completed successfully 2020-04-04 11:57:59Z [overcloud.AllNodesDeploySteps.NetworkerDeployment_Step2]: CREATE_COMPLETE state changed 2020-04-04 11:59:11Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.2]: SIGNAL_IN_PROGRESS Signal: deployment 991e0577-2894-4c7f-a3aa-69a3d6382991 succeeded 2020-04-04 11:59:12Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.2]: CREATE_COMPLETE state changed 2020-04-04 11:59:12Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment d374fcc0-d638-451f-9673-02ea4f3Heat Stack create failed. Heat Stack create failed. 4809b succeeded 2020-04-04 11:59:14Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1]: CREATE_COMPLETE state changed 2020-04-04 11:59:56Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment f2009ff2-f666-4334-81f8-72280cd27212 failed (2) 2020-04-04 11:59:56Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2020-04-04 11:59:56Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2020-04-04 11:59:57Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step2]: CREATE_FAILED Error: resources.ControllerDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2020-04-04 11:59:57Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2020-04-04 11:59:58Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2020-04-04 11:59:58Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 Stack overcloud CREATE_FAILED overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: f2009ff2-f666-4334-81f8-72280cd27212 status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | ... "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_vip]/Pacemaker::Constraint::Colocation[storage_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.3.11-haproxy-bundle]: Skipping because of failed dependencies", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_mgmt_vip]/Pacemaker::Constraint::Order[storage_mgmt_vip-then-haproxy]/Pcmk_constraint[order-ip-172.17.4.15-haproxy-bundle]: Skipping because of failed dependencies", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_mgmt_vip]/Pacemaker::Constraint::Colocation[storage_mgmt_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.4.15-haproxy-bundle]: Skipping because of failed dependencies" ] } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/8f4cc510-125a-47d9-ae3b-f1ef2e00f1ff_playbook.retry PLAY RECAP ********************************************************************* localhost : ok=6 changed=2 unreachable=0 failed=1 (truncated, view all with --long) deploy_stderr: |