Bug 1656540

Summary: Scale-out fails because kolla can't copy/delete /etc/hosts
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-tripleo-heat-templatesAssignee: Alex Schultz <aschultz>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: agurenko, aschultz, dhill, gkumar, jschluet, mburns, tb-rhel, uemit.seren
Target Milestone: z5Keywords: TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-8.0.7-4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-14 13:55:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1578849    
Bug Blocks:    

Description David Hill 2018-12-05 17:55:09 UTC 
Description of problem:
Scale-out fails because kolla can't copy/delete /etc/hosts and the following commands solves the issue:

# mv /var/lib/config-data/puppet-generated/nova_libvirt/etc/hosts /root/
# docker restart <CONTAINER>


Dec  5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506.
Dec  5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container 5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32.
Dec  5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs
Dec  5 07:48:35 overcloud-compute-6 systemd: Started libcontainer container b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e.
Dec  5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/default/neutron-server
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/default/neutron-server to /etc/default/neutron-server
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugin.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugin.ini to /etc/neutron/plugin.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugins/ml2/ml2_conf.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/ml2_conf.ini to /etc/neutron/plugins/ml2/ml2_conf.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/plugins/ml2/openvswitch_agent.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/plugins/ml2/openvswitch_agent.ini to /etc/neutron/plugins/ml2/openvswitch_agent.ini
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/neutron/neutron.conf
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts
Dec  5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error:
Dec  5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last):
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 411, in main
Dec  5 07:48:35 overcloud-compute-6 journal:    execute_config_strategy(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Dec  5 07:48:35 overcloud-compute-6 journal:    copy_config(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Dec  5 07:48:35 overcloud-compute-6 journal:    config_file.copy()
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 150, in copy
Dec  5 07:48:35 overcloud-compute-6 journal:    self._merge_directories(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    os.path.join(dest, to_copy))
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    self._copy_file(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file
Dec  5 07:48:35 overcloud-compute-6 journal:    self._delete_path(dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path
Dec  5 07:48:35 overcloud-compute-6 journal:    os.remove(path)
Dec  5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts'
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/ceilometer/ceilometer.conf
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ceilometer/ceilometer.conf to /etc/ceilometer/ceilometer.conf
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts
Dec  5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error:
Dec  5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last):
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 411, in main
Dec  5 07:48:35 overcloud-compute-6 journal:    execute_config_strategy(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Dec  5 07:48:35 overcloud-compute-6 journal:    copy_config(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Dec  5 07:48:35 overcloud-compute-6 journal:    config_file.copy()
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 150, in copy
Dec  5 07:48:35 overcloud-compute-6 journal:    self._merge_directories(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    os.path.join(dest, to_copy))
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    self._copy_file(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file
Dec  5 07:48:35 overcloud-compute-6 journal:    self._delete_path(dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path
Dec  5 07:48:35 overcloud-compute-6 journal:    os.remove(path)
Dec  5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts'
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.57236363Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 does not exist\\none or more of the container deletions failed\\n\""
Dec  5 07:48:35 overcloud-compute-6 journal: + sudo -E kolla_set_configs
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.593965735Z" level=warning msg="1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 cleanup: failed to unmount secrets: invalid argument"
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Validating config file
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Copying service configuration files
Dec  5 07:48:35 overcloud-compute-6 journal: INFO:__main__:Deleting /etc/hosts
Dec  5 07:48:35 overcloud-compute-6 journal: ERROR:__main__:Unexpected error:
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.619112098Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32 does not exist\\none or more of the container deletions failed\\n\""
Dec  5 07:48:35 overcloud-compute-6 journal: Traceback (most recent call last):
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 411, in main
Dec  5 07:48:35 overcloud-compute-6 journal:    execute_config_strategy(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Dec  5 07:48:35 overcloud-compute-6 journal:    copy_config(config)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Dec  5 07:48:35 overcloud-compute-6 journal:    config_file.copy()
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 150, in copy
Dec  5 07:48:35 overcloud-compute-6 journal:    self._merge_directories(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    os.path.join(dest, to_copy))
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories
Dec  5 07:48:35 overcloud-compute-6 journal:    self._copy_file(source, dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.636019771Z" level=warning msg="5e61f97dd8734c521e6c27019f775245b186660d36e91e5a78b46c25ac0f8c32 cleanup: failed to unmount secrets: invalid argument"
Dec  5 07:48:35 overcloud-compute-6 journal:    self._delete_path(dest)
Dec  5 07:48:35 overcloud-compute-6 journal:  File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path
Dec  5 07:48:35 overcloud-compute-6 journal:    os.remove(path)
Dec  5 07:48:35 overcloud-compute-6 journal: OSError: [Errno 16] Device or resource busy: '/etc/hosts'
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.674045206Z" level=error msg="containerd: deleting container" error="exit status 1: \"container b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e does not exist\\none or more of the container deletions failed\\n\""
Dec  5 07:48:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:48:35.699836063Z" level=warning msg="b6cee236185ae2420a0e8f5fb80a140780a3555a2b20cd2177e9d6f16009745e cleanup: failed to unmount secrets: invalid argument"
Dec  5 07:49:05 overcloud-compute-6 dockerd-current: time="2018-12-05T07:49:05.466423867Z" level=warning msg="Health check for container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 error: No such exec instance 'eaec1179bc6c1935db6b066c86cce12cafbd699a958e0c7ef3a5833e0626b439' found in daemon"
Dec  5 07:49:35 overcloud-compute-6 dockerd-current: time="2018-12-05T07:49:35.466700124Z" level=warning msg="Health check for container 1527d2c24aaa331cc6326dbb2eb4b3124f4e764a42bcb61aded4fefe43125506 error: No such exec instance '33759b3798e76834208b7bb79506151eef1ddeb3b996fc417e967fa3b8d9935c' found in daemon"

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Alex Schultz 2018-12-05 22:40:01 UTC 
Please check that the host time is synced and the bios is set to utc.
Comment 2 Alex Schultz 2018-12-05 23:09:44 UTC 
What version of openstack-tripleo-heat-templates are being used?  This was fixed in openstack-tripleo-heat-templates-8.0.7-4.el7ost
Comment 3 David Hill 2018-12-05 23:46:11 UTC 
Thanks,  Customer have openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch so I let him know this should be fixed by updating.
Comment 4 David Hill 2018-12-05 23:46:50 UTC 
And yes, the clocks were all in sync as I found the other BZ where syncing the clocks solved this issue.   In this case, it didn't but we'll see after updating.
Comment 5 nict-tb 2018-12-06 02:01:39 UTC 
In some cases you also need:

# mv /var/lib/config-data/puppet-generated/neutron/etc/hosts /root/neutron_hosts
# mv /var/lib/config-data/puppet-generated/crond/etc/hosts /root/crond_hosts
# mv /var/lib/config-data/puppet-generated/ceilometer/etc/hosts /root/ceilometer_hosts
Comment 6 Alex Schultz 2018-12-06 14:47:32 UTC 
The other issue that causes this is if the bios TZ is not in UTC.
Comment 7 Uemit Seren 2018-12-23 08:15:40 UTC 
Was this backported to Pike (OSP12) ? 
We are running into the same issue with Pike and I checked in the ntp.yml and we don't have the fix in our version of openstack-tripleo-heat-templates-7.0.12-12.el7ost.noarch.
Comment 8 Alex Schultz 2019-01-02 15:25:00 UTC 
Unlikely backported to OSP12. OSP12 is also now EOL.
Comment 9 Lon Hohberger 2019-01-03 11:46:01 UTC 
According to our records, this should be resolved by openstack-tripleo-heat-templates-8.0.7-4.el7ost.  This build is available now.
Comment 12 errata-xmlrpc 2019-03-14 13:55:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0448