Description of problem: Scale-out fails because kolla can't copy/delete /etc/hosts and the following commands solves the issue: # mv /var/lib/config-data/puppet-generated/nova_libvirt/etc/hosts /root/ # docker restart <CONTAINER> Dec 5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506. Dec 5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container 5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32. Dec 5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs Dec 5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e. Dec 5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/default/neutron-server Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/default/neutron-server to /etc/default/neutron-server Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugin.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugin.ini to /etc/neutron/plugin.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugins/ml2/ml2_conf.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugins/ml2/openvswitch_agent.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini to /etc/neutron/plugins/ml2/openvswitch_agent.ini Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/neutron.conf Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts Dec 5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error: Dec 5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last): Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 411, in main Dec 5 07:48:35 overcloud-compute-6 journal: execute_config_strategy(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy Dec 5 07:48:35 overcloud-compute-6 journal: copy_config(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 306, in copy_config Dec 5 07:48:35 overcloud-compute-6 journal: config_file.copy() Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 150, in copy Dec 5 07:48:35 overcloud-compute-6 journal: self._merge_directories(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: os.path.join(dest, to_copy)) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: self._copy_file(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file Dec 5 07:48:35 overcloud-compute-6 journal: self._delete_path(dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path Dec 5 07:48:35 overcloud-compute-6 journal: os.remove(path) Dec 5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts' Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/ceilometer/ceilometer.conf Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/ceilometer.conf to /etc/ceilometer/ceilometer.conf Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts Dec 5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error: Dec 5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last): Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 411, in main Dec 5 07:48:35 overcloud-compute-6 journal: execute_config_strategy(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy Dec 5 07:48:35 overcloud-compute-6 journal: copy_config(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 306, in copy_config Dec 5 07:48:35 overcloud-compute-6 journal: config_file.copy() Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 150, in copy Dec 5 07:48:35 overcloud-compute-6 journal: self._merge_directories(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: os.path.join(dest, to_copy)) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: self._copy_file(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file Dec 5 07:48:35 overcloud-compute-6 journal: self._delete_path(dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path Dec 5 07:48:35 overcloud-compute-6 journal: os.remove(path) Dec 5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts' Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.57236363Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 does not exist\\none or more of the container deletions failed\\n\"" Dec 5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.593965735Z" level=warning msg="1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 cleanup: failed to unmount secrets: invalid argument" Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files Dec 5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts Dec 5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error: Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.619112098Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32 does not exist\\none or more of the container deletions failed\\n\"" Dec 5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last): Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 411, in main Dec 5 07:48:35 overcloud-compute-6 journal: execute_config_strategy(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy Dec 5 07:48:35 overcloud-compute-6 journal: copy_config(config) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 306, in copy_config Dec 5 07:48:35 overcloud-compute-6 journal: config_file.copy() Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 150, in copy Dec 5 07:48:35 overcloud-compute-6 journal: self._merge_directories(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: os.path.join(dest, to_copy)) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories Dec 5 07:48:35 overcloud-compute-6 journal: self._copy_file(source, dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.636019771Z" level=warning msg="5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32 cleanup: failed to unmount secrets: invalid argument" Dec 5 07:48:35 overcloud-compute-6 journal: self._delete_path(dest) Dec 5 07:48:35 overcloud-compute-6 journal: File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path Dec 5 07:48:35 overcloud-compute-6 journal: os.remove(path) Dec 5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts' Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.674045206Z" level=error msg="containerd: deleting container" error="exit status 1: \"container b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e does not exist\\none or more of the container deletions failed\\n\"" Dec 5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.699836063Z" level=warning msg="b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e cleanup: failed to unmount secrets: invalid argument" Dec 5 07:49:05 overcloud-compute-6 dockerd-current: time="2018-12-05T07:49:05.466423867Z" level=warning msg="Health check for container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 error: No such exec instance 'eaec1179bc6c1935db6b066c86cce12cafbd699a958e0c7ef3a5833e0626b439' found in daemon" Dec 5 07:49:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:49:35.466700124Z" level=warning msg="Health check for container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 error: No such exec instance '33759b3798e76834208b7bb79506151eef1ddeb3b996fc417e967fa3b8d9935c' found in daemon" Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Please check that the host time is synced and the bios is set to utc.
What version of openstack-tripleo-heat-templates are being used? This was fixed in openstack-tripleo-heat-templates-8.0.7-4.el7ost
Thanks, Customer have openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch so I let him know this should be fixed by updating.
And yes, the clocks were all in sync as I found the other BZ where syncing the clocks solved this issue. In this case, it didn't but we'll see after updating.
In some cases you also need: # mv /var/lib/config-data/puppet-generated/neutron/etc/hosts /root/neutron_hosts # mv /var/lib/config-data/puppet-generated/crond/etc/hosts /root/crond_hosts # mv /var/lib/config-data/puppet-generated/ceilometer/etc/hosts /root/ceilometer_hosts
The other issue that causes this is if the bios TZ is not in UTC.
Was this backported to Pike (OSP12) ? We are running into the same issue with Pike and I checked in the ntp.yml and we don't have the fix in our version of openstack-tripleo-heat-templates-7.0.12-12.el7ost.noarch.
Unlikely backported to OSP12. OSP12 is also now EOL.
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.0.7-4.el7ost. This build is available now.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0448