Description of problem: It seems that Overcloud deploy fails once galera and messaging node composable roles are being deployed on Pacemaker Remote nodes with support of containers. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Deploy overcloud with 3 galera and 3 messaging nodes which are Pacemaker remote nodes based. Actual results: The overcloud deploy fail on AllNodesDeploySteps.DatabaseDeployment_Step2.0: Additional info: Output from "openstack stack failures list overcloud": "Error running ['docker', 'run', '--name', 'mysql_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=mysql_init_bundle', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 1, \"command\": [\"/bin/bash\", \"-c\", \"cp -a /tmp/puppet-etc/* /etc/puppet; echo \\'{\\\\\"step\\\\\": 2}\\' > /etc/puppet/hieradata/docker.json; FACTER_uuid=docker puppet apply --tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,galera_ready,mysql_database,mysql_grant,mysql_user -v -e \\'include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::database::mysql_bundle\\'\"], \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro\", \"/dev/shm:/dev/shm:rw\", \"/var/lib/mysql:/var/lib/mysql:rw\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-mariadb-docker:20171004.1\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro', '--volume=/dev/shm:/dev/shm:rw', '--volume=/var/lib/mysql:/var/lib/mysql:rw', '192.168.24.1:8787/rhosp12/openstack-mariadb-docker:20171004.1', '/bin/bash', '-c', 'cp -a /tmp/puppet-etc/* /etc/puppet; echo \\'{\"step\": 2}\\' > /etc/puppet/hieradata/docker.json; FACTER_uuid=docker puppet apply --tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,galera_ready,mysql_database,mysql_grant,mysql_user -v -e \\'include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::database::mysql_bundle\\'']. [1]", "stdout: \u001b[0;32mInfo: Loading facts\u001b[0m", "\u001b[0;32mInfo: Loading facts\u001b[0m", "\u001b[mNotice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend\u001b[0m", "\u001b[mNotice: Compiled catalog for database-0.localdomain in environment production in 3.86 seconds\u001b[0m", "\u001b[0;32mInfo: Applying configuration version '1508171078'\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/root/.my.cnf]/ensure: defined content as '{md5}70bfd721763f95ceb016a9a2a1c5b5a2'\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/File[/etc/sysconfig/clustercheck]/ensure: defined content as '{md5}b181285cb14a8e6e152d2d7070a48844'\u001b[0m", "\u001b[0;32mInfo: Computing checksum on file /etc/my.cnf.d/galera.cnf\u001b[0m", "\u001b[0;32mInfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf.d/galera.cnf to puppet with sum f434e1d5766874c7b9ed08d0c66904ca\u001b[0m", "\u001b[mNotice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}f434e1d5766874c7b9ed08d0c66904ca' to '{md5}dd551613bf7d533ae3a3ce23535046ab'\u001b[0m", "\u001b[0;32mInfo: Class[Mysql::Server::Config]: Unscheduling all events on Class[Mysql::Server::Config]\u001b[0m", "\u001b[mNotice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Class[Mysql::Server::Installdb]: Unscheduling all events on Class[Mysql::Server::Installdb]\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-database-0]/Pcmk_property[property-database-0-galera-role]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Pacemaker::Property[galera-role-database-0]: Unscheduling all events on Pacemaker::Property[galera-role-database-0]\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-database-1]/Pcmk_property[property-database-1-galera-role]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Pacemaker::Property[galera-role-database-1]: Unscheduling all events on Pacemaker::Property[galera-role-database-1]\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-database-2]/Pcmk_property[property-database-2-galera-role]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Pacemaker::Property[galera-role-database-2]: Unscheduling all events on Pacemaker::Property[galera-role-database-2]\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Pacemaker::Resource::Bundle[galera-bundle]: Unscheduling all events on Pacemaker::Resource::Bundle[galera-bundle]\u001b[0m", "\u001b[mNotice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]/ensure: created\u001b[0m", "\u001b[0;32mInfo: Pacemaker::Resource::Ocf[galera]: Unscheduling all events on Pacemaker::Resource::Ocf[galera]\u001b[0m", "\u001b[0;32mInfo: Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle]: Unscheduling all events on Class[Tripleo::Profile::Pacemaker::Database::Mysql_bundle]\u001b[0m", "\u001b[0;32mInfo: Creating state file /var/lib/puppet/state/state.yaml\u001b[0m", "stderr: Failed to get D-Bus connection: Operation not permitted", "\u001b[1;33mWarning: Facter: Could not retrieve fact='erl_ssl_path', resolution='<anonymous>': undefined method `gsub!' for false:FalseClass\u001b[0m", "\u001b[1;33mWarning: Undefined variable 'deploy_config_name'; ", " (file & line not available)\u001b[0m", "\u001b[1;33mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql_bundle.pp\", 94]:[\"unknown\", 1]", " (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')\u001b[0m", "\u001b[1;33mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 97]:[\"unknown\", 1]", "\u001b[1;33mWarning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/aodh/manifests/db/mysql.pp\", 58]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/database/mysql.pp\", 166]", "\u001b[1;33mWarning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /etc/puppet/modules/nova/manifests/db/mysql.pp:53:28\u001b[0m", "\u001b[1;33mWarning: ModuleLoader: module 'panko' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules", "\u001b[1;33mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\"/etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp\", 43]:", "\u001b[1;31mError: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]\u001b[0m", "\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from notrun to 0 failed: /usr/bin/clustercheck >/dev/null returned 1 instead of one of [0]\u001b[0m", "\u001b[1;31mError: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m" Pacemaker status at the time of failure: Cluster name: tripleo_cluster Stack: corosync Current DC: controller-0 (version 1.1.16-12.el7_4.2-94ff4df) - partition with quorum Last updated: Mon Oct 16 17:29:52 2017 Last change: Mon Oct 16 16:28:06 2017 by redis-bundle-0 via crm_attribute on controller-0 18 nodes configured 42 resources configured Online: [ controller-0 controller-1 controller-2 ] RemoteOnline: [ database-0 database-1 database-2 messaging-0 messaging-1 messaging-2 ] GuestOnline: [ rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ] Full list of resources: database-0 (ocf::pacemaker:remote): Started controller-0 database-1 (ocf::pacemaker:remote): Started controller-1 database-2 (ocf::pacemaker:remote): Started controller-2 messaging-0 (ocf::pacemaker:remote): Started controller-0 messaging-1 (ocf::pacemaker:remote): Started controller-1 messaging-2 (ocf::pacemaker:remote): Started controller-2 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 Docker container set: galera-bundle [192.168.24.1:8787/rhosp12/openstack-mariadb-docker:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped galera-bundle-1 (ocf::heartbeat:galera): Stopped galera-bundle-2 (ocf::heartbeat:galera): Stopped Docker container set: redis-bundle [192.168.24.1:8787/rhosp12/openstack-redis-docker:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave controller-2 ip-192.168.24.8 (ocf::heartbeat:IPaddr2): Started controller-0 ip-10.0.0.107 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.1.13 (ocf::heartbeat:IPaddr2): Started controller-2 ip-172.17.1.15 (ocf::heartbeat:IPaddr2): Started controller-0 ip-172.17.3.19 (ocf::heartbeat:IPaddr2): Started controller-1 ip-172.17.4.11 (ocf::heartbeat:IPaddr2): Started controller-2 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp12/openstack-haproxy-docker:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started controller-2 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
*** This bug has been marked as a duplicate of bug 1507910 ***