rhel-osp-director: 7.2 - HA overcloud deployment with netiso fails Error: resources.ControllerNodesPostDeployment.resources.ControllerPostPuppet.resources.ControllerPostPuppetRestartDeployment.resources[0]: Deployment to server failed" Environment: instack-undercloud-2.1.2-34.el7ost.noarch openstack-swift-container-2.3.0-2.el7ost.noarch redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch openstack-heat-engine-2015.1.2-2.el7ost.noarch openstack-ceilometer-common-2015.1.2-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-79.el7ost.noarch openstack-heat-templates-0-0.7.20150605git.el7ost.noarch openstack-selinux-0.6.43-1.el7ost.noarch openstack-tempest-kilo-20151020.1.el7ost.noarch openstack-neutron-ml2-2015.1.2-2.el7ost.noarch python-django-openstack-auth-1.2.0-5.el7ost.noarch openstack-swift-proxy-2.3.0-2.el7ost.noarch openstack-tripleo-puppet-elements-0.0.1-5.el7ost.noarch openstack-ceilometer-collector-2015.1.2-1.el7ost.noarch openstack-ironic-common-2015.1.2-2.el7ost.noarch openstack-ceilometer-api-2015.1.2-1.el7ost.noarch openstack-ironic-api-2015.1.2-2.el7ost.noarch openstack-keystone-2015.1.2-2.el7ost.noarch openstack-tuskar-ui-extras-0.0.4-2.el7ost.noarch openstack-glance-2015.1.2-1.el7ost.noarch openstack-heat-api-2015.1.2-2.el7ost.noarch openstack-ceilometer-central-2015.1.2-1.el7ost.noarch openstack-puppet-modules-2015.1.8-30.el7ost.noarch openstack-nova-compute-2015.1.2-4.el7ost.noarch openstack-nova-conductor-2015.1.2-4.el7ost.noarch openstack-swift-object-2.3.0-2.el7ost.noarch openstack-dashboard-2015.1.2-3.el7ost.noarch openstack-heat-common-2015.1.2-2.el7ost.noarch openstack-tuskar-0.4.18-5.el7ost.noarch openstack-neutron-openvswitch-2015.1.2-2.el7ost.noarch openstack-nova-api-2015.1.2-4.el7ost.noarch openstack-nova-console-2015.1.2-4.el7ost.noarch openstack-neutron-common-2015.1.2-2.el7ost.noarch openstack-neutron-2015.1.2-2.el7ost.noarch openstack-heat-api-cfn-2015.1.2-2.el7ost.noarch openstack-ironic-conductor-2015.1.2-2.el7ost.noarch openstack-swift-2.3.0-2.el7ost.noarch openstack-nova-novncproxy-2015.1.2-4.el7ost.noarch openstack-swift-account-2.3.0-2.el7ost.noarch openstack-tuskar-ui-0.4.0-5.el7ost.noarch openstack-utils-2014.2-1.el7ost.noarch openstack-tripleo-common-0.0.1.dev6-4.git49b57eb.el7ost.noarch openstack-heat-api-cloudwatch-2015.1.2-2.el7ost.noarch openstack-ceilometer-notification-2015.1.2-1.el7ost.noarch python-openstackclient-1.0.3-3.el7ost.noarch openstack-ceilometer-alarm-2015.1.2-1.el7ost.noarch openstack-swift-plugin-swift3-1.7-3.el7ost.noarch openstack-dashboard-theme-2015.1.2-3.el7ost.noarch openstack-ironic-discoverd-1.1.0-8.el7ost.noarch openstack-nova-scheduler-2015.1.2-4.el7ost.noarch openstack-nova-cert-2015.1.2-4.el7ost.noarch openstack-nova-common-2015.1.2-4.el7ost.noarch openstack-tripleo-image-elements-0.9.6-10.el7ost.noarch Steps to reproduce: Attempt to deploy HA overcloud with network isolation. Result: Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates Stack failed with status: Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerPostPuppet.resources.ControllerPostPuppetRestartDeployment.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 ERROR: openstack Heat Stack create failed. | resource_status_reason | Error: resources.ControllerNodesPostDeployment.resources.ControllerPostPuppet.resources.ControllerPostPuppetRestartDeployment.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 | Seeing this error in os-collect-config: Nov 20 16:32:15 overcloud-controller-0.localdomain os-collect-config[4484]: cemaker::Stonith/Exec[Disable STONITH]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Loadbalancer/Haproxy::Listen[redis]/Concat::Fragment[redis_listen_block]/File[/var/lib/puppet/concat/_etc_haproxy_haproxy.cfg/fragments/20-redis-00_redis_listen_block]/ensure: defined content as '{md5}cd28fbbf8e50285de05d3378cc28f58e'\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy::Config/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy::Config/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]: Triggered 'refresh' from 43 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy::Config/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: content changed '{md5}1f337186b0e1ba5ee82760cb437fb810' to '{md5}5e72f14d4e4467d247cecd526bb2586b'\u001b[0m\n\u001b[mNotice: /File[/etc/haproxy/haproxy.cfg]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: Finished catalog run in 48.16 seconds\u001b[0m\n", "deploy_stderr": "\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0} Nov 20 16:32:15 overcloud-controller-0.localdomain os-collect-config[4484]: [2015-11-20 16:32:15,186] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Nov 20 16:32:15 overcloud-controller-0.localdomain os-collect-config[4484]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Nov 20 16:36:10 overcloud-controller-0.localdomain os-collect-config[4484]: ]/Pcmk_constraint[colo-ip-192.168.120.10-haproxy-clone]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Colocation[public_vip-with-haproxy]/Pcmk_constraint[colo-ip-192.168.200.180-haproxy-clone]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[public_vip-then-haproxy]/Exec[Creating order constraint public_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Colocation[internal_api_vip-with-haproxy]/Pcmk_constraint[colo-ip-192.168.100.11-haproxy-clone]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[redis_vip-then-haproxy]/Exec[Creating order constraint redis_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[storage_vip-then-haproxy]/Exec[Creating order constraint storage_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: Finished catalog run in 129.93 seconds\u001b[0m\n", "deploy_stderr": "\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0} Nov 20 16:36:10 overcloud-controller-0.localdomain os-collect-config[4484]: [2015-11-20 16:36:10,677] (heat-config) [INFO] Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Nov 20 16:36:10 overcloud-controller-0.localdomain os-collect-config[4484]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Expected results: Successful deployment.
Created attachment 1097345 [details] /var/log dir from one controller.
In /var/log/messages of the attached log dir i can see: Nov 20 17:11:11 localhost os-collect-config: httpd never started after 200 seconds, which is: https://bugzilla.redhat.com/show_bug.cgi?id=1285363 *** This bug has been marked as a duplicate of bug 1285363 ***