[OSP12][openstack containers]: overlcoud deployment fails due to Error running docker u'mysql_bootstrap'. Environment: ------------ openstack-tripleo-image-elements-6.0.0-2.el7ost.noarch openstack-tripleo-puppet-elements-6.0.0-3.el7ost.noarch openstack-tripleo-ui-3.1.0-9.el7ost.noarch openstack-tripleo-validations-5.5.0-1.el7ost.noarch python-tripleoclient-6.1.0-6.el7ost.noarch openstack-tripleo-heat-templates-6.0.0-12.el7ost.noarch puppet-tripleo-6.3.0-12.el7ost.noarch openstack-tripleo-common-6.0.0-8.el7ost.noarch openstack-puppet-modules-10.0.0-1.el7ost.noarch Steps: ------ Attempt to deploy osp12 with containers : 1 controller , 1 compute overcloud containers - IMAGE_TAG:2017-05-16.6 Failure view : -------------- eohjms2pfwag.BlockStorageContainersDeployment_Step2]: CREATE_COMPLETE state changed 2017-05-30 16:35:10Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ObjectStorageContainersDeployment_Step2]: CREATE_COMPLETE state changed 2017-05-30 16:35:20Z [0]: SIGNAL_IN_PROGRESS Signal: deployment 0a4daf08-d963-4ad6-bcea-4e830b4962d7 succeeded 2017-05-30 16:35:20Z [0]: CREATE_COMPLETE state changed 2017-05-30 16:35:20Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ComputeContainersDeployment_Step2]: CREATE_COMPLETE Stack CREATE completed successfully 2017-05-30 16:35:20Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ComputeContainersDeployment_Step2]: CREATE_COMPLETE state changed 2017-05-30 16:36:48Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ControllerContainersDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment 717520f7-28c4-457d-9976-050fb0e19eed failed (1) 2017-05-30 16:36:48Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ControllerContainersDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2017-05-30 16:36:48Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ControllerContainersDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 2017-05-30 16:36:49Z [overcloud-AllNodesDeploySteps-eohjms2pfwag.ControllerContainersDeployment_Step2]: CREATE_FAILED Error: resources.ControllerContainersDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-05-30 16:36:49Z [overcloud-AllNodesDeploySteps-eohjms2pfwag]: UPDATE_FAILED Error: resources.ControllerContainersDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-05-30 16:36:50Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerContainersDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-05-30 16:36:50Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerContainersDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 Stack overcloud UPDATE_FAILED overcloud.AllNodesDeploySteps.ControllerContainersDeployment_Step2.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 717520f7-28c4-457d-9976-050fb0e19eed status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | ... Support MariaDB development by buying support/new features from MariaDB Corporation Ab. You can contact us about this at sales. Alternatively consider joining our community based development effort: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/ 170530 16:35:45 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 170530 16:35:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 170530 16:35:47 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended f9db0f59a4b62d3d848e262dc112a85d5c845f58bace7a3a09abaadba2cdfe2d (truncated, view all with --long) deploy_stderr: | INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/lib/mysql 170530 16:35:45 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170530 16:35:45 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 51 ... 170530 16:35:45 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170530 16:35:45 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 59 ... Heat Stack update failed. Heat Stack update failed. (undercloud) [stack@undercloud-0 ~]$ Debug: ------ (undercloud) [stack@undercloud-0 ~]$ heat resource-list overcloud -n 5 | grep -v COMPLETE WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | AllNodesDeploySteps | e84a037b-9e1a-4e4e-86ec-4ea0352dead7 | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-05-30T19:10:27Z | overcloud | | ControllerContainersDeployment_Step2 | 231a8324-5461-43b9-8770-decf520b1459 | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-05-30T19:24:01Z | overcloud-AllNodesDeploySteps-ux32behsp5wx | | 0 | 1e72dd11-58e4-410a-9ac4-1b4ed0c336b7 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-05-30T19:33:19Z | overcloud-AllNodesDeploySteps-ux32behsp5wx-ControllerContainersDeployment_Step2-chlxeayxopxy | +----------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ (undercloud) [stack@undercloud-0 ~]$ heat deployment-show 1e72dd11-58e4-410a-9ac4-1b4ed0c336b7 WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "62a09adc-b8d9-4ead-beb3-f248b4c682dd", "config_id": "b121c6a0-dc41-4291-ac92-c364e2f6ee2f", "output_values": { "deploy_stdout": "e536c252be3eea431235a7b8d2c2ab5fb56da985309d9f6e60092e5d44b7b850\n\ne98d319a9a4a721b9ff6098456fff2380f11fd13edc182751e18ae96be64e372\n\nInstalling MariaDB/MySQL system tables in '/var/lib/mysql' ...\nOK\nFilling help tables...\nOK\n\nTo start mysqld at boot time you have to copy\nsupport-files/mysql.server to the right place for your system\n\nPLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !\nTo do so, start the server, then issue the following commands:\n\n'/usr/bin/mysqladmin' -u root password 'new-password'\n'/usr/bin/mysqladmin' -u root -h controller-0.localdomain password 'new-password'\n\nAlternatively you can run:\n'/usr/bin/mysql_secure_installation'\n\nwhich will also give you the option of removing the test\ndatabases and anonymous user created by default. This is\nstrongly recommended for production servers.\n\nSee the MariaDB Knowledgebase at http://mariadb.com/kb or the\nMySQL manual for more instructions.\n\nYou can start the MariaDB daemon with:\ncd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql'\n\nYou can test the MariaDB daemon with mysql-test-run.pl\ncd '/usr/mysql-test' ; perl mysql-test-run.pl\n\nPlease report any problems at http://mariadb.org/jira\n\nThe latest information about MariaDB is available at http://mariadb.org/.\nYou can find additional information about the MySQL part at:\nhttp://dev.mysql.com\nSupport MariaDB development by buying support/new features from MariaDB\nCorporation Ab. You can contact us about this at sales.\nAlternatively consider joining our community based development effort:\nhttp://mariadb.com/kb/en/contributing-to-the-mariadb-project/\n\n170530 19:33:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.\n170530 19:33:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql\n170530 19:34:00 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended\n\nfc2c2078b9388a160c40609555b10f21454771c1de598f338a5d7bbd9442831e\n", "deploy_stderr": "INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Writing out command to execute\nINFO:__main__:Setting permission for /var/lib/mysql\n170530 19:33:58 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295\n170530 19:33:58 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 51 ...\n170530 19:33:58 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295\n170530 19:33:58 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 59 ...\n", "deploy_status_code": 1 }, "creation_time": "2017-05-30T19:33:20Z", "updated_time": "2017-05-30T19:34:59Z", "input_values": {}, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "1e72dd11-58e4-410a-9ac4-1b4ed0c336b7" } (undercloud) [stack@undercloud-0 ~]$ ssh controller --> from /var/log/messages : --------------------------------------------- May 30 15:34:58 localhost os-collect-config: [2017-05-30 19:34:57,917] (heat-config) [ERROR] Error running ['docker', 'run', '--name', u'mysql_bootstrap', '--label', u'deploy_stack_id=overcloud-AllNodesDeploySteps-ux32behsp5wx-ControllerContainersDeployment_Step2-chlxeayxopxy/231a8324-5461-43b9-8770-decf520b1459', '--label', u'deploy_resource_name=0', '--label', u'config_id=b121c6a0-dc41-4291-ac92-c364e2f6ee2f', '--label', u'container_name=mysql_bootstrap', '--label', 'managed_by=docker-cmd', u'--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', u'--env=KOLLA_BOOTSTRAP=True', u'--env=KOLLA_KUBERNETES=True', u'--env=DB_ROOT_PASSWORD=KJn7W2oKUS', u'--net=host', u'--volume=/var/lib/kolla/config_files/mysql.json:/var/lib/kolla/config_files/config.json', u'--volume=/var/lib/config-data/mysql/etc/:/etc/:ro', u'--volume=/etc/localtime:/etc/localtime:ro', u'--volume=/etc/hosts:/etc/hosts:ro', u'--volume=/var/lib/mysql:/var/lib/mysql', u'--volume=/var/log/containers/mysql:/var/log/mariadb', u'docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb-docker:2017-05-16.6', u'bash', u'-c', u'test -e /var/lib/mysql/mysql || kolla_start']. [1]
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
Reproduced.
Looking at the logs of the respective container: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/run/redis Running command: '/usr/bin/redis-server /etc/redis.conf' [root@controller-0 ~]# docker logs ad3bbffe8422 INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/lib/mysql Installing MariaDB/MySQL system tables in '/var/lib/mysql' ... 170530 19:33:58 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170530 19:33:58 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 51 ... OK Filling help tables... 170530 19:33:58 [Warning] option 'open_files_limit': unsigned value 18446744073709551615 adjusted to 4294967295 170530 19:33:58 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 59 ... OK To start mysqld at boot time you have to copy support-files/mysql.server to the right place for your system PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER ! To do so, start the server, then issue the following commands: '/usr/bin/mysqladmin' -u root password 'new-password' '/usr/bin/mysqladmin' -u root -h controller-0.localdomain password 'new-password' Alternatively you can run: '/usr/bin/mysql_secure_installation' which will also give you the option of removing the test databases and anonymous user created by default. This is strongly recommended for production servers. See the MariaDB Knowledgebase at http://mariadb.com/kb or the MySQL manual for more instructions. You can start the MariaDB daemon with: cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql' You can test the MariaDB daemon with mysql-test-run.pl cd '/usr/mysql-test' ; perl mysql-test-run.pl Please report any problems at http://mariadb.org/jira The latest information about MariaDB is available at http://mariadb.org/. You can find additional information about the MySQL part at: http://dev.mysql.com Support MariaDB development by buying support/new features from MariaDB Corporation Ab. You can contact us about this at sales. Alternatively consider joining our community based development effort: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/ 170530 19:33:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. 170530 19:33:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 170530 19:34:00 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
Reproduced but I think this is the same bug as - https://bugzilla.redhat.com/show_bug.cgi?id=1450370 I will check workaround
workaround works, deployments successfully passed this step [heat-admin@overcloud-controller-0 ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 29b42ea17f52 192.168.24.1:8787/rhosp12/openstack-mariadb-docker:workaround "kolla_start" 5 minutes ago Up 5 minutes mysql 3cad69053e69 192.168.24.1:8787/rhosp12/openstack-mongodb-docker:2017-05-16.6 "kolla_start" 5 minutes ago Up 5 minutes mongodb a0d2e0ea43d3 192.168.24.1:8787/rhosp12/openstack-rabbitmq-docker:2017-05-16.6 "kolla_start" 9 minutes ago Up 9 minutes rabbitmq f675619a3b19 192.168.24.1:8787/rhosp12/openstack-memcached-docker:2017-05-16.6 "/bin/bash -c 'source" 9 minutes ago Up 9 minutes memcached 5eae27c40562 192.168.24.1:8787/rhosp12/openstack-redis-docker:2017-05-16.6 "kolla_start" 9 minutes ago Up 9 minutes redis ... So I think this bug is duplicate
Yeah, I confirmed that is duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1450370 because I received next error: Stack overcloud CREATE_FAILED overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: dd7dca6f-2262-49b7-be50-9af6bb225f95 status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 deploy_stdout: | ... Notice: /Stage[main]/Horizon::Wsgi::Apache/Apache::Vhost[horizon_vhost]/File[/var/www/]/seluser: seluser changed 'unconfined_u' to 'system_u' Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[/var/www/cgi-bin/cinder]/ensure: created Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/File[cinder_wsgi]/ensure: defined content as '{md5}2edefdbaaa05fe52ec75b860834ad713' Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[/var/www/cgi-bin/heat]/ensure: created Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/File[heat_api_cloudwatch_wsgi]/ensure: defined content as '{md5}d30e0d9c58374044881079f8dd8132e8' Notice: /Stage[main]/Horizon::Wsgi::Apache/Apache::Vhost[horizon_vhost]/Concat[10-horizon_vhost.conf]/File[/etc/httpd/conf.d/10-horizon_vhost.conf]/ensure: defined content as '{md5}0ca70c5e8c08d92ec6395f3a8cbb2cd6' Notice: /Stage[main]/Cinder::Wsgi::Apache/Openstacklib::Wsgi::Apache[cinder_wsgi]/Apache::Vhost[cinder_wsgi]/Concat[10-cinder_wsgi.conf]/File[/etc/httpd/conf.d/10-cinder_wsgi.conf]/ensure: defined content as '{md5}46c882c35280d67d3ce3ca209b9852f8' Notice: /Stage[main]/Heat::Wsgi::Apache_api_cloudwatch/Heat::Wsgi::Apache[api_cloudwatch]/Openstacklib::Wsgi::Apache[heat_api_cloudwatch_wsgi]/Apache::Vhost[heat_api_cloudwatch_wsgi]/Concat[10-heat_api_cloudwatch_wsgi.conf]/File[/etc/httpd/conf.d/10-heat_api_cloudwatch_wsgi.conf]/ensure: defined content as '{md5}79bb23f2ffd5811cb0d4ccfa137020e2' Notice: /Stage[main]/Apache::Service/Service[httpd]/ensure: ensure changed 'stopped' to 'running' Notice: Applied catalog in 668.44 seconds (truncated, view all with --long) deploy_stderr: | ... (file & line not available) Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/pacemaker/haproxy_with_vip.pp", 62]: (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation') Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications. Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Failed to call refresh: Command exceeded timeout Error: /Stage[main]/Cinder::Db::Sync/Exec[cinder-manage db_sync]: Command exceeded timeout Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: Command exceeded timeout Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Command exceeded timeout (truncated, view all with --long) Heat Stack create failed. Heat Stack create failed. and this error is https://bugzilla.redhat.com/show_bug.cgi?id=1450370
I'm closing this one as a duplicate of bug 1450370. *** This bug has been marked as a duplicate of bug 1450370 ***