Description of problem: 2017-09-16 11:15:07Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1]: CREATE_FAILED Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2017-09-16 11:15:07Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0]: CREATE_FAILED CREATE aborted 2017-09-16 11:15:07Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.2]: CREATE_FAILED CREATE aborted 2017-09-16 11:15:07Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_FAILED Resource CREATE failed: Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 2017-09-16 11:15:08Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_FAILED Error: resources.ControllerDeployment_Step1.resources[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2017-09-16 11:15:08Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerDeployment_Step1.resources[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2017-09-16 11:15:09Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step1.resources[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 2017-09-16 11:15:09Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step1.resources[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2 Stack overcloud CREATE_FAILED overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 406b791c-b485-4348-a7e5-04780dbe6d53 status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | ... ], "changed": false, "failed": true, "failed_when_result": true } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/a878dc21-dfc6-4fb7-9f4f-f4c314b14514_playbook.retry PLAY RECAP ********************************************************************* localhost : ok=8 changed=1 unreachable=0 failed=1 (truncated, view all with --long) deploy_stderr: | overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 53a3adc7-6b03-42d8-8bb3-1dbb27cc97b4 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | None deploy_stderr: | None overcloud.AllNodesDeploySteps.ControllerDeployment_Step1.2: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 63afac1c-42b0-4865-8747-045513e7fe40 status: CREATE_FAILED status_reason: | CREATE aborted deploy_stdout: | None deploy_stderr: | None Heat Stack create failed. Heat Stack create failed. from stack failures list --long overcloud , "Error running ['docker', 'run', '--name', 'mysql_bootstrap', '--label', 'config_id=tripleo_step1', '--label', 'container_name=mysql_bootstrap', '--label', 'managed_by=paunch', '--label', 'config_data={\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"KOLLA_BOOTSTRAP=True\", \"KOLLA_KUBERNETES=True\", \"DB_MAX_TIMEOUT=60\", \"DB_CLUSTERCHECK_PASSWORD=XtXtGQK3tADgWZAY6spWwfb8d\", \"DB_ROOT_PASSWORD=rW3Lf5Avqt\"], \"start_order\": 1, \"command\": [\"bash\", \"-ecx\", \"if [ -e /var/lib/mysql/mysql ]; then exit 0; fi\\\\nkolla_start\\\\nmysqld_safe --skip-networking --wsrep-on=OFF --wsrep-provider=none &\\\\ntimeout ${DB_MAX_TIMEOUT} /bin/bash -c \\'until mysqladmin -uroot -p\\\\\"${DB_ROOT_PASSWORD}\\\\\" ping 2>/dev/null; do sleep 1; done\\'\\\\nmysql -uroot -p\\\\\"${DB_ROOT_PASSWORD}\\\\\" -e \\\\\"CREATE USER \\'clustercheck\\'@\\'localhost\\' IDENTIFIED BY \\'${DB_CLUSTERCHECK_PASSWORD}\\';\\\\\"\\\\nmysql -uroot -p\\\\\"${DB_ROOT_PASSWORD}\\\\\" -e \\\\\"GRANT PROCESS ON *.* TO \\'clustercheck\\'@\\'localhost\\' WITH GRANT OPTION;\\\\\"\\\\ntimeout ${DB_MAX_TIMEOUT} mysqladmin -uroot -p\\\\\"${DB_ROOT_PASSWORD}\\\\\" shutdown\"], \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro\", \"/var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/mysql:/var/lib/mysql\"], \"image\": \"192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-08.3\", \"detach\": false, \"net\": \"host\"}', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_KUBERNETES=True', '--env=DB_MAX_TIMEOUT=60', '--env=DB_CLUSTERCHECK_PASSWORD=XtXtGQK3tADgWZAY6spWwfb8d', '--env=DB_ROOT_PASSWORD=rW3Lf5Avqt', '--net=host', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json', '--volume=/var/lib/config-data/puppet-generated/mysql/:/var/lib/kolla/config_files/src:ro', '--volume=/var/lib/mysql:/var/lib/mysql', '192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-08.3', 'bash', '-ecx', 'if [ -e /var/lib/mysql/mysql ]; then exit 0; fi\\nkolla_start\\nmysqld_safe --skip-networking --wsrep-on=OFF --wsrep-provider=none &\\ntimeout ${DB_MAX_TIMEOUT} /bin/bash -c \\'until mysqladmin -uroot -p\"${DB_ROOT_PASSWORD}\" ping 2>/dev/null; do sleep 1; done\\'\\nmysql -uroot -p\"${DB_ROOT_PASSWORD}\" -e \"CREATE USER \\'clustercheck\\'@\\'localhost\\' IDENTIFIED BY \\'${DB_CLUSTERCHECK_PASSWORD}\\';\"\\nmysql -uroot -p\"${DB_ROOT_PASSWORD}\" -e \"GRANT PROCESS ON *.* TO \\'clustercheck\\'@\\'localhost\\' WITH GRANT OPTION;\"\\ntimeout ${DB_MAX_TIMEOUT} mysqladmin -uroot -p\"${DB_ROOT_PASSWORD}\" shutdown']. [1]", "stdout: Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...", "OK", "Filling help tables...", "To start mysqld at boot time you have to copy", "support-files/mysql.server to the right place for your system", "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !", "To do so, start the server, then issue the following commands:", "'/usr/bin/mysqladmin' -u root password 'new-password'", "'/usr/bin/mysqladmin' -u root -h controller-2.redhat.local password 'new-password'", "Alternatively you can run:", "'/usr/bin/mysql_secure_installation'", "which will also give you the option of removing the test", "databases and anonymous user created by default. This is", "strongly recommended for production servers.", "See the MariaDB Knowledgebase at http://mariadb.com/kb or the", "MySQL manual for more instructions.", "You can start the MariaDB daemon with:", "cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql'", "You can test the MariaDB daemon with mysql-test-run.pl", "cd '/usr/mysql-test' ; perl mysql-test-run.pl", "Please report any problems at http://mariadb.org/jira", "The latest information about MariaDB is available at http://mariadb.org/.", "You can find additional information about the MySQL part at:", "http://dev.mysql.com", "Support MariaDB development by buying support/new features from MariaDB", "Corporation Ab. You can contact us about this at sales.", "Alternatively consider joining our community based development effort:", "http://mariadb.com/kb/en/contributing-to-the-mariadb-project/", "170916 10:31:32 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.", "170916 10:31:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql", "170916 10:31:32 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.K0j0Tk' --pid-file='/var/lib/mysql/controller-2.redhat.local-recover.pid'", "170916 10:31:34 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1", "170916 10:31:34 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended", "stderr: + '[' -e /var/lib/mysql/mysql ']'", "+ kolla_start", "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json", "INFO:__main__:Validating config file", "INFO:__main__:Kolla config strategy set to: COPY_ALWAYS", "INFO:__main__:Copying service configuration files", "INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets", "INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets", "INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf", "INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck", "INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf", "INFO:__main__:Writing out command to execute", "170916 10:31:29 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295", "170916 10:31:30 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295", "stdout: 05c56213f06ef9033747ac852a3bf252a86a666be80f0f75c755f5a79ba30e79" ], "changed": false, "failed": true, "failed_when_result": true } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/1784a2a2-63e4-4a4a-bc83-32e73ff7068a_playbook.retry [heat-admin@controller-0 ~]$ sudo docker ps -a |grep "Exited (1)" 48ecc6e3c43f 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:2017-09-08.3 "bash -ecx 'if [ -e /" 4 minutes ago Exited (1) 3 minutes ago mysql_bootstrap Version-Release number of selected component (if applicable): OSP12 How reproducible: osp12 puddle - 2017-09-08.3 openstack-tripleo-validations-7.3.1-0.20170831052729.67faa39.el7ost.noarch openstack-selinux-0.8.9-0.1.el7ost.noarch openstack-nova-placement-api-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-swift-object-2.15.2-0.20170824165102.c54c6b3.el7ost.noarch openstack-swift-proxy-2.15.2-0.20170824165102.c54c6b3.el7ost.noarch openstack-ironic-api-9.1.1-0.20170824135903.d783dff.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170901051303.0rc1.el7ost.noarch openstack-tempest-16.1.1-0.20170830101230.e70e0fe.el7ost.noarch openstack-mistral-engine-5.0.1-0.20170830120805.4bc0950.el7ost.noarch openstack-neutron-openvswitch-11.0.1-0.20170831212231.d6f8c44.el7ost.noarch openstack-heat-engine-9.0.1-0.20170830131248.2b94474.el7ost.noarch python-openstackclient-lang-3.12.0-0.20170821150739.f67ebce.el7ost.noarch openstack-nova-common-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-swift-account-2.15.2-0.20170824165102.c54c6b3.el7ost.noarch openstack-heat-common-9.0.1-0.20170830131248.2b94474.el7ost.noarch python-openstackclient-3.12.0-0.20170821150739.f67ebce.el7ost.noarch puppet-openstacklib-11.3.1-0.20170825142820.18ee919.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch openstack-tripleo-common-containers-7.5.1-0.20170831015950.el7ost.noarch python-openstacksdk-0.9.17-0.20170821143340.7946243.el7ost.noarch openstack-tripleo-image-elements-7.0.0-0.20170830150703.526772d.el7ost.noarch openstack-mistral-common-5.0.1-0.20170830120805.4bc0950.el7ost.noarch openstack-tripleo-ui-7.3.1-0.20170830131652.f61181a.el7ost.noarch openstack-nova-api-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-nova-conductor-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-nova-compute-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-heat-api-9.0.1-0.20170830131248.2b94474.el7ost.noarch openstack-ironic-common-9.1.1-0.20170824135903.d783dff.el7ost.noarch openstack-mistral-executor-5.0.1-0.20170830120805.4bc0950.el7ost.noarch openstack-keystone-12.0.1-0.20170830123737.6a67918.el7ost.noarch openstack-neutron-ml2-11.0.1-0.20170831212231.d6f8c44.el7ost.noarch openstack-heat-api-cfn-9.0.1-0.20170830131248.2b94474.el7ost.noarch openstack-zaqar-5.0.1-0.20170830120218.9207f7e.el7ost.noarch puppet-openstack_extras-11.3.1-0.20170825142718.352987f.el7ost.noarch openstack-glance-15.0.0-0.20170830130905.9820166.el7ost.noarch openstack-swift-container-2.15.2-0.20170824165102.c54c6b3.el7ost.noarch openstack-neutron-11.0.1-0.20170831212231.d6f8c44.el7ost.noarch openstack-ironic-conductor-9.1.1-0.20170824135903.d783dff.el7ost.noarch openstack-tripleo-common-7.5.1-0.20170831015950.el7ost.noarch openstack-mistral-api-5.0.1-0.20170830120805.4bc0950.el7ost.noarch openstack-nova-scheduler-16.0.1-0.20170830161812.cdf08b7.el7ost.noarch openstack-tripleo-puppet-elements-7.0.0-0.20170831100659.2094778.el7ost.noarch openstack-neutron-common-11.0.1-0.20170831212231.d6f8c44.el7ost.noarch openstack-ironic-inspector-6.0.1-0.20170824132804.0e72dcb.el7ost.noarch Steps to Reproduce: http://etherpad.corp.redhat.com/osp12-internal-SSL-using-freeIPA Actual results: Deployment failed Expected results: Additional info: docker logs for mysql_bootstrap contaiiner [heat-admin@controller-1 ~]$ sudo docker logs mysql_bootstrap + '[' -e /var/lib/mysql/mysql ']' + kolla_start INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck INFO:__main__:Copying /var/lib/kolla/config_files/src/root/.my.cnf to /root/.my.cnf INFO:__main__:Writing out command to execute Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 170916 11:14:03 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 OK Filling help tables... 170916 11:14:04 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 OK To start mysqld at boot time you have to copy support-files/mysql.server to the right place for your system PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! To do so, start the server, then issue the following commands: '/usr/bin/mysqladmin' -u root password 'new-password' '/usr/bin/mysqladmin' -u root -h controller-1.redhat.local password 'new-password' Alternatively you can run: '/usr/bin/mysql_secure_installation' which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. See the MariaDB Knowledgebase at http://mariadb.com/kb or the MySQL manual for more instructions. You can start the MariaDB daemon with: cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql' You can test the MariaDB daemon with mysql-test-run.pl cd '/usr/mysql-test' ; perl mysql-test-run.pl Please report any problems at http://mariadb.org/jira The latest information about MariaDB is available at http://mariadb.org/. You can find additional information about the MySQL part at: http://dev.mysql.com Support MariaDB development by buying support/new features from MariaDB Corporation Ab. You can contact us about this at sales. Alternatively consider joining our community based development effort: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/ 170916 11:14:06 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 170916 11:14:06 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 170916 11:14:06 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.G8k9aF' --pid-file='/var/lib/mysql/controller-1.redhat.local-recover.pid' 170916 11:14:08 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 170916 11:14:08 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
Would this be related? https://bugs.launchpad.net/tripleo/+bug/1717545
(In reply to Juan Antonio Osorio from comment #2) > Would this be related? https://bugs.launchpad.net/tripleo/+bug/1717545 I don't think we have any issue with downstream python and pymysql. But that could be a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1483643 Artem, would you be able to test https://review.openstack.org/#/c/504049/ on your deploy or give me access to it so we can check together?
Artem confirmed this is a duplicate of #1483643. *** This bug has been marked as a duplicate of bug 1483643 ***
yep