Bug 1592505
Summary: | OVN - Deploying OSP13 fails in the overcloud deployment phase | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Daniel Alvarez Sanchez <dalvarez> |
Component: | openstack-tripleo | Assignee: | James Slagle <jslagle> |
Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> |
Severity: | urgent | Docs Contact: | |
Priority: | unspecified | ||
Version: | 13.0 (Queens) | CC: | aschultz, dalvarez, dciabrin, dpeacock, mburns |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-06-20 16:33:05 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Daniel Alvarez Sanchez
2018-06-18 17:31:47 UTC
Please can you reproduce this in a non-infrared env; that is to say a regular deployment. Looks like an env problem at first glance. It's a freshly provisioned BM with CentOS. You want me to use Director instead? What kind of env problem are you thinking of? Thanks! Through docker logs I see the following: INFO:__main__:Creating directory /etc/rabbitmq/ssl INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rabbitmq/inetrc to /etc/rabbitmq/inetrc INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rabbitmq/rabbitmq-env.conf to /etc/rabbitmq/rabbitmq-env.conf INFO:__main__:Deleting /etc/rabbitmq/rabbitmq.config INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rabbitmq/rabbitmq.config to /etc/rabbitmq/rabbitmq.config INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rabbitmq/rabbitmqadmin.conf to /etc/rabbitmq/rabbitmqadmin.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/security/limits.d/rabbitmq-server.conf to /etc/security/limits.d/rabbitmq-server.conf INFO:__main__:Creating directory /etc/systemd/system/rabbitmq-server.service.d INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/systemd/system/rabbitmq-server.service.d/limits.conf to /etc/systemd/system/rabbitmq-server.service.d/limits.conf INFO:__main__:Deleting /etc/hosts ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 411, in main execute_config_strategy(config) File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy copy_config(config) File "/usr/local/bin/kolla_set_configs", line 306, in copy_config config_file.copy() File "/usr/local/bin/kolla_set_configs", line 150, in copy self._merge_directories(source, dest) File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories os.path.join(dest, to_copy)) File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories self._copy_file(source, dest) File "/usr/local/bin/kolla_set_configs", line 75, in _copy_file self._delete_path(dest) File "/usr/local/bin/kolla_set_configs", line 108, in _delete_path os.remove(path) OSError: [Errno 16] Device or resource busy: '/etc/hosts' [root@controller-2 ~]# docker logs 7e3686b1f5ef INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Copying /dev/null to /etc/libqb/force-filesystem-sockets INFO:__main__:Setting permission for /etc/libqb/force-filesystem-sockets INFO:__main__:Deleting /etc/my.cnf.d/galera.cnf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/galera.cnf to /etc/my.cnf.d/galera.cnf ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 411, in main execute_config_strategy(config) File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy copy_config(config) File "/usr/local/bin/kolla_set_configs", line 306, in copy_config config_file.copy() File "/usr/local/bin/kolla_set_configs", line 150, in copy self._merge_directories(source, dest) File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories os.path.join(dest, to_copy)) File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories os.path.join(dest, to_copy)) File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories os.path.join(dest, to_copy)) File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories self._set_properties(source, dest) File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties self._set_properties_from_file(source, dest) File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file shutil.copystat(source, dest) File "/usr/lib64/python2.7/shutil.py", line 98, in copystat os.utime(dst, (st.st_atime, st.st_mtime)) OSError: [Errno 30] Read-only file system: '/etc/pki/ca-trust/extracted' For some reason on that deployment, docker-puppet.py seems to copy file /etc/hosts and directory /etc/pki into every service's /var/lib/config-data/puppet-generated/{service}, which is unexpected and invalid. [...] /var/lib/config-data/puppet-generated/redis/etc/pki /var/lib/config-data/puppet-generated/redis/etc/hosts /var/lib/config-data/puppet-generated/heat/etc/pki /var/lib/config-data/puppet-generated/heat/etc/hosts /var/lib/config-data/puppet-generated/nova/etc/pki /var/lib/config-data/puppet-generated/nova/etc/hosts /var/lib/config-data/puppet-generated/glance_api/etc/pki /var/lib/config-data/puppet-generated/glance_api/etc/hosts /var/lib/config-data/puppet-generated/rabbitmq/etc/pki /var/lib/config-data/puppet-generated/rabbitmq/etc/hosts /var/lib/config-data/puppet-generated/heat_api_cfn/etc/pki /var/lib/config-data/puppet-generated/heat_api_cfn/etc/hosts /var/lib/config-data/puppet-generated/keystone/etc/pki /var/lib/config-data/puppet-generated/keystone/etc/hosts [...] Not only HA-specific. Trying to figure out why this is happening It seems that when the docker-puppet-{service} container were ran, some timestamps from the container image were more recent than the timezone the container were started in (so file that appeared to be in the future), so when docker-puppet.sh touches the file /var/lib/config-data/{service}.origin_of_time, so some files in the container still had a more recent timestamps, and when puppet finished and docker-puppet.sh ended up copying spurious files in /var/lib/config-data/puppet-generated/{service}. So subsequent kolla_init failed and deployment went in error. This seems like another occurrence of https://bugzilla.redhat.com/show_bug.cgi?id=1578849 (In reply to Damien Ciabrini from comment #9) > This seems like another occurrence of > https://bugzilla.redhat.com/show_bug.cgi?id=1578849 Either a timezone issue due to NTP setting, or the consumption of that container happened in a timezone where the container's file in the image appeared to be in the future. Marking this as a dupe of Bug 1578849 as we'll be addressing time sync issues with that bug. That should resolve the underlying cause for this as well. *** This bug has been marked as a duplicate of bug 1578849 *** |