Created attachment 1402545 [details]
stack failures list long and rabbimq_init container logs

Description of problem:
During controller replacement procedure

after Manual intervention
10.4.3. Manual Intervention - https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/html-single/director_installation_and_usage/#sect-Replacing_Controller_Nodes

step "Open a new terminal and re-run the overcloud deployment command to continue the stack update: "

after re-run of overcloud_deploy.sh

failures list - http://pastebin.test.redhat.com/560459

pcs status - http://pastebin.test.redhat.com/560463

"Error running ['docker', 'run', '--name', 'rabbitmq_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=rabbitmq_init_bundle', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 0, \"command\": [\"/docker_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready\", \"include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle\", \"--debug\"], \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/docker-config-scripts/docker_puppet_apply.sh:/docker_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro\", \"/dev/shm:/dev/shm:rw\", \"/bin/true:/bin/epmd\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-rabbitmq:2018-02-23.1\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/docker-config-scripts/docker_puppet_apply.sh:/docker_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro', '--volume=/dev/shm:/dev/shm:rw', '--volume=/bin/true:/bin/epmd', '192.168.24.1:8787/rhosp12/openstack-rabbitmq:2018-02-23.1', '/docker_puppet_apply.sh', '2', 'file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready', 'include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle', '--debug']


On failed controller

559f0b9c6f59        192.168.24.1:8787/rhosp12/openstack-rabbitmq:2018-02-23.1      "/docker_puppet_apply"   2 hours ago         Exited (1) 2 hours ago                       rabbitmq_init_bundle

docker logs in attachment

Version-Release number of selected component (if applicable):
osp12
openstack-rabbitmq:2018-02-23.1 

How reproducible:
Always

Steps to Reproduce:
1.Deploy OSP12
2.Try to replace controller using official manual
3.After re-deployment after manual intervention deploy will fail

Actual results:
Stack was updated

Expected results:
UPDATE_FAILED

Additional info:
in attachment