Description of problem: We have two OpenStack deployments, one is a deployed successfully and one is ALWAYS failing on: [root@controller-0 ~]# paunch debug --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_1.json --user 0 --container mysql_bootstrap --action run + '[' -e /var/lib/mysql/mysql ']' + echo -e '\n[mysqld]\nwsrep_provider=none' + sudo -u mysql -E kolla_start INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sysconfig/clustercheck to /etc/sysconfig/clustercheck INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/hosts.old to /etc/hosts.old INFO:__main__:Deleting /etc/hosts ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 411, in main execute_config_strategy(config) File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy copy_config(config) File "/usr/local/bin/kolla_set_configs", line 306, in copy_config config_file.copy() File "/usr/local/bin/kolla_set_configs", line 150, in copy self._merge_directories(source, dest) File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories os.path.join(dest, to_copy)) File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories self._copy_file(source, dest) File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file self._delete_path(dest) File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path os.remove(path) OSError: [Errno 16] Device or resource busy: '/etc/hosts' [root@controller-0 ~]# At some point the deployment tries to spawn a container that wants to delte /etc/hosts. When we removed the file: /var/lib/config-data/puppet-generated/mysql/etc/hosts We seen similar bugzilla open in upstream on tripleo-ci but wasn't reproducible after latest packages https://bugs.launchpad.net/tripleo/+bug/1709689 Version-Release number of selected component (if applicable): (undercloud) [stack@undercloud-0 share]$ rpm -qa | grep tripleo openstack-tripleo-common-containers-7.6.3-7.el7ost.noarch openstack-tripleo-heat-templates-7.0.3-17.el7ost.noarch puppet-tripleo-7.4.3-11.el7ost.noarch openstack-tripleo-common-7.6.3-7.el7ost.noarch openstack-tripleo-ui-7.4.3-4.el7ost.noarch openstack-tripleo-validations-7.4.2-1.el7ost.noarch openstack-tripleo-puppet-elements-7.0.1-2.el7ost.noarch python-tripleoclient-7.3.3-6.el7ost.noarch openstack-tripleo-image-elements-7.0.1-1.el7ost.noarch How reproducible: [root@controller-0 ~]# stat /etc/hosts File: ‘/etc/hosts’ Size: 3066 Blocks: 8 IO Block: 4096 regular file Device: fc02h/64514d Inode: 460 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:net_conf_t:s0 Access: 2017-12-13 11:31:03.713390986 +0000 Modify: 2017-12-13 11:31:03.396390986 +0000 Change: 2017-12-13 09:43:21.758750643 +0000 Birth: - [root@controller-0 ~]# stat /var/lib/config-data/puppet-generated/mysql/etc/hosts File: ‘/var/lib/config-data/puppet-generated/mysql/etc/hosts’ Size: 3066 Blocks: 8 IO Block: 4096 regular file Device: fc02h/64514d Inode: 84085094 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:container_var_lib_t:s0 Access: 2017-12-13 07:13:17.634695686 +0000 Modify: 2017-12-13 11:31:03.000000000 +0000 Change: 2017-12-13 07:13:17.472696989 +0000 Birth: - [root@controller-0 ~]# diff -u /var/lib/config-data/puppet-generated/mysql/etc/hosts /etc/hosts [root@controller-0 ~]# [root@controller-0 ~]# docker images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE 192.168.24.1:8787/rhosp12/openstack-nova-placement-api 12.0-20171129.1 sha256:971aa7036177a34b0e1682fe2bf40ad4fcfec9e094ede09b0a16a4f63b685399 6b0351163ff0 13 days ago 731.2 MB 192.168.24.1:8787/rhosp12/openstack-aodh-api 12.0-20171129.1 sha256:ab5532dcd5abf6bbe2368bbf07b6cb7d4dc607b48d5644bb191ee047ad1a6b92 b61a4387f865 13 days ago 677.5 MB 192.168.24.1:8787/rhosp12/openstack-glance-api 12.0-20171129.1 sha256:cc3ae76155b007b6051d2e2d9ca5a5f61b4814a1511334beeb70f01c126f1dee 944b3f491c2e 13 days ago 831.1 MB 192.168.24.1:8787/rhosp12/openstack-nova-api 12.0-20171129.1 sha256:e18b7a69916ecd8a22b040aedd0650fb49bb94ad2e60587995e5f47afd28d2df f9581a5c3353 13 days ago 731.2 MB 192.168.24.1:8787/rhosp12/openstack-panko-api 12.0-20171129.1 sha256:a520eb69b44ae1c55bfdcb3aaac08f09020cc8252f6d33c4a832d386aada7b40 e2f08ddec8a7 13 days ago 691.6 MB 192.168.24.1:8787/rhosp12/openstack-heat-api-cfn 12.0-20171129.1 sha256:deba8bddc6dd12e8d466a9ed20fba3576fee53ede3b06308ce892637f5f51b1b bf7cbae96cfe 13 days ago 680.6 MB 192.168.24.1:8787/rhosp12/openstack-ceilometer-central 12.0-20171129.1 sha256:2a395a8537b0b24d024a2d9a64bef16ef0285cd0ea3403ea08b9b46b04c3fd38 e39afc496a86 13 days ago 699.4 MB 192.168.24.1:8787/rhosp12/openstack-heat-api 12.0-20171129.1 sha256:8a1b758aff396d913bdcd86ca8cba124cc2d4da978820e8c586e46d38fe2686b 5aa8b4f0cb00 13 days ago 680.6 MB 192.168.24.1:8787/rhosp12/openstack-heat-engine 12.0-20171129.1 sha256:4f919df1c6ff11bb5f155880c5c501f661ecf9f837d4ca092a5767f94cf26991 a672f2979f82 13 days ago 680.6 MB 192.168.24.1:8787/rhosp12/openstack-keystone 12.0-20171129.1 sha256:c914a1c62812d9e722d58f1499d75ee94d7298d33bfac2f6251169e749eb9b0c bd45cfae8ac1 13 days ago 659.8 MB 192.168.24.1:8787/rhosp12/openstack-gnocchi-api 12.0-20171129.1 sha256:3d8ef18f345d6450bf73bc8e5ae3e8c99c0fffa660497f98a5bb77d0920b5121 bf089c78dec3 13 days ago 958.5 MB 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server 12.0-20171129.1 sha256:f7bc1e48a98b23d500ad055f67cff7e1075637022e7b974932eb6d734e247e52 78f1a90c4b78 13 days ago 693 MB 192.168.24.1:8787/rhosp12/openstack-horizon 12.0-20171129.1 sha256:b4ec23a40a3d1a58dc4559b6273fd09d126a8c402856bf05932cc6af048ddeaf 07444b6c11fd 13 days ago 740.4 MB 192.168.24.1:8787/rhosp12/openstack-mariadb 12.0-20171129.1 sha256:789aa346a4865bb06031fddbf75e26781be9cc67693884a0c466cfabd7d1cb1f 6d6f0bc78831 13 days ago 904.9 MB 192.168.24.1:8787/rhosp12/openstack-mariadb pcmklatest sha256:789aa346a4865bb06031fddbf75e26781be9cc67693884a0c466cfabd7d1cb1f 6d6f0bc78831 13 days ago 904.9 MB 192.168.24.1:8787/rhosp12/openstack-rabbitmq 12.0-20171129.1 sha256:0964dc9ffe28527d61da871d2bbf1557d464023015bbc57afefa41d1af2f2d9a 716b358a3921 13 days ago 808.8 MB 192.168.24.1:8787/rhosp12/openstack-rabbitmq pcmklatest sha256:0964dc9ffe28527d61da871d2bbf1557d464023015bbc57afefa41d1af2f2d9a 716b358a3921 13 days ago 808.8 MB 192.168.24.1:8787/rhosp12/openstack-redis 12.0-20171129.1 sha256:02d7de41b66e798cd181414bf60726d037a5171ce6a5d94439ac59dfe40b25ad 03bca6ccbf7f 13 days ago 774.1 MB 192.168.24.1:8787/rhosp12/openstack-redis pcmklatest sha256:02d7de41b66e798cd181414bf60726d037a5171ce6a5d94439ac59dfe40b25ad 03bca6ccbf7f 13 days ago 774.1 MB 192.168.24.1:8787/rhosp12/openstack-haproxy 12.0-20171129.1 sha256:b92e563e12b4e39aad11bf19465d5d63c8d2ccfe1e2a52401ccd539576cd9d2a 2a9767dddf79 13 days ago 774.4 MB 192.168.24.1:8787/rhosp12/openstack-haproxy pcmklatest sha256:b92e563e12b4e39aad11bf19465d5d63c8d2ccfe1e2a52401ccd539576cd9d2a 2a9767dddf79 13 days ago 774.4 MB 192.168.24.1:8787/rhosp12/openstack-memcached 12.0-20171129.1 sha256:c8608246810c195ceaba3c30f730386777c968a624ca5e6e3113f72bca58f082 5a3a6ef051d9 13 days ago 384.5 MB 192.168.24.1:8787/rhosp12/openstack-cron 12.0-20171129.1 sha256:40165ab10859029a825dcfca13bc08bde783645819157ec91aa1c5e8fceb656b 24ac13538657 13 days ago 341.1 MB Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1367219 [details] Failures after second run
[stack@undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash timeout 100m openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /home/stack/virt/docker-images.yaml \ --log-file overcloud_deployment_87.log
Created attachment 1367220 [details] deploy command
Created attachment 1367221 [details] Docker extra logs Addind docker extra logs
Created attachment 1367222 [details] Install log
Created attachment 1367224 [details] attaching Event list Note that this is from second run, not sure it failed same.
There is a time difference between undercloud and overcloud. undercloud time is not synced with NTP server and showing older times
After Syncing time and re-deploying whole environment cleared the issue.