Hide Forgot
Description of problem: OSP11 -> OSP12 upgrade: swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_rsync, swift_account_replicator, swift_proxy containers are restarting after upgrade: [root@controller-0 heat-admin]# docker ps | grep swift | grep Restart 72d5388ac7b0 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (1) 52 minutes ago swift_object_expirer ce808ac19a31 192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (1) 52 minutes ago swift_container_replicator 8e2824bb8aa1 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (1) 52 minutes ago swift_object_replicator f46bfd80ca5c 192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (10) 52 minutes ago swift_rsync 6706b3a3cd84 192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (1) 52 minutes ago swift_account_replicator 251154c79f13 192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1 "kolla_start" About an hour ago Restarting (1) 52 minutes ago swift_proxy Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-7.0.0-0.20170718190543.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 2. Upgrade to OSP12: #!/bin/bash timeout 100m openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \ -e /home/stack/docker-osp12.yaml \ Actual results: swift containers are in Restarting status. Expected results: swift containers are up Additional info: [root@controller-0 heat-admin]# docker logs --tail 10 swift_proxy ring_name='container') File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__ self._reload(force=True) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload ring_data = RingData.load(self.serialized_path) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load gz_file = GzipFile(filename, 'rb') File "/usr/lib64/python2.7/gzip.py", line 94, in __init__ fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz' [root@controller-0 heat-admin]# docker logs --tail 10 swift_object_expirer ring_name='container') File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__ self._reload(force=True) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload ring_data = RingData.load(self.serialized_path) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load gz_file = GzipFile(filename, 'rb') File "/usr/lib64/python2.7/gzip.py", line 94, in __init__ fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz' [root@controller-0 heat-admin]# docker logs --tail 10 swift_container_replicator self.ring = ring.Ring(swift_dir, ring_name=self.server_type) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__ self._reload(force=True) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload ring_data = RingData.load(self.serialized_path) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load gz_file = GzipFile(filename, 'rb') File "/usr/lib64/python2.7/gzip.py", line 94, in __init__ fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz' [root@controller-0 heat-admin]# docker logs --tail 10 swift_object_replicator INFO:__main__:Deleting /etc/swift/swift.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf INFO:__main__:Deleting /etc/xinetd.d/rsync INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.d/rsync to /etc/xinetd.d/rsync INFO:__main__:Deleting /etc/rsyncd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf INFO:__main__:Deleting /etc/xinetd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.conf to /etc/xinetd.conf INFO:__main__:Writing out command to execute Running command: '/usr/bin/swift-object-replicator /etc/swift/object-server.conf' [root@controller-0 heat-admin]# docker logs --tail 10 swift_rsync INFO:__main__:Deleting /etc/swift/swift.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf INFO:__main__:Deleting /etc/xinetd.d/rsync INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.d/rsync to /etc/xinetd.d/rsync INFO:__main__:Deleting /etc/rsyncd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf INFO:__main__:Deleting /etc/xinetd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.conf to /etc/xinetd.conf INFO:__main__:Writing out command to execute Running command: '/usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf' [root@controller-0 heat-admin]# docker logs --tail 10 swift_account_replicator self.ring = ring.Ring(swift_dir, ring_name=self.server_type) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__ self._reload(force=True) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload ring_data = RingData.load(self.serialized_path) File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load gz_file = GzipFile(filename, 'rb') File "/usr/lib64/python2.7/gzip.py", line 94, in __init__ fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb') IOError: [Errno 2] No such file or directory: '/etc/swift/account.ring.gz'
The missing file is there during 11, so my guess here is that during the migration of concurrent services into containers (and therefore cleaning them from the node) the file is removed which causes the IO error. Temp assigning to Marios for our next triaging meeting tomorrow.
There is a pretty much similar bug reported for osp12 fresh deployment: bug 1474866 so this might be a duplicate.
There are no ring files in /var/lib/config-data/puppet-generated/swift/etc/swift/ [root@controller-0 heat-admin]# ls -l /var/lib/config-data/puppet-generated/swift/etc/swift/ total 28 drwxr-xr-x. 2 42445 42445 6 Jul 31 17:34 account-server -rw-r--r--. 1 42445 42445 656 Jul 31 17:34 account-server.conf drwxr-xr-x. 2 42445 42445 6 Jul 31 17:34 container-server -rw-r--r--. 1 42445 42445 745 Jul 31 17:34 container-server.conf -rw-r-----. 1 root 42445 527 Jul 31 17:34 object-expirer.conf drwxr-xr-x. 2 42445 42445 6 Jul 31 17:34 object-server -rw-r--r--. 1 42445 42445 679 Jul 31 17:34 object-server.conf -rw-r-----. 1 root 42445 4211 Jul 31 17:34 proxy-server.conf -rw-r-----. 1 42445 42445 106 Jul 31 17:34 swift.conf but these are present in /etc/swift: [root@controller-0 heat-admin]# ls -l /etc/swift/ total 84 -rw-r--r--. 1 swift swift 9277 Jul 31 15:18 account.builder -rw-r--r--. 1 root root 249 Jul 31 15:18 account.ring.gz drwxr-xr-x. 2 root root 6 Jul 26 07:01 account-server -rw-r--r--. 1 swift swift 656 Jul 31 15:28 account-server.conf drwxr-xr-x. 2 swift swift 4096 Jul 31 15:18 backups -rw-r--r--. 1 swift swift 9277 Jul 31 15:18 container.builder -rw-r-----. 1 root swift 1415 Jul 26 06:53 container-reconciler.conf -rw-r--r--. 1 root root 250 Jul 31 15:18 container.ring.gz drwxr-xr-x. 2 root root 6 Jul 26 07:01 container-server -rw-r--r--. 1 swift swift 745 Jul 31 15:28 container-server.conf -rw-r--r--. 1 swift swift 9277 Jul 31 15:18 object.builder -rw-r-----. 1 root swift 529 Jul 31 15:24 object-expirer.conf -rw-r--r--. 1 root root 247 Jul 31 15:18 object.ring.gz drwxr-xr-x. 2 root root 6 Jul 26 07:01 object-server -rw-r--r--. 1 swift swift 655 Jul 31 15:28 object-server.conf drwxr-xr-x. 2 root root 6 Jul 26 07:01 proxy-server -rw-r-----. 1 root swift 4809 Jul 31 15:28 proxy-server.conf -rw-r-----. 1 swift swift 106 Jul 31 15:14 swift.conf
After running: cp /etc/swift/*gz /var/lib/config-data/puppet-generated/swift/etc/swift/ for container in swift_proxy swift_object_expirer swift_container_replicator swift_object_replicator swift_rsync swift_account_replicator; do docker stop $container; docker start $container; done only the swift_rsync container keeps restarting.
It seems that some manual steps are needed to configure swift: http://tripleo.org/install/advanced_deployment/deploy_swift.html Which in fact, it is related to the workaround Marius posted, copy the files manually and restart the service. But I wonder, couldn't this be automated?
updated to include stable/pike review upstream
[root@controller-0 heat-admin]# docker ps | grep swift 213996a6ca16 registry:5000/rhosp12/openstack-swift-account-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_account_server 2cbb553fcc67 registry:5000/rhosp12/openstack-swift-container-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_container_auditor 8eb669ac1b56 registry:5000/rhosp12/openstack-swift-proxy-server-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_object_expirer a74274d06532 registry:5000/rhosp12/openstack-swift-object-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_object_updater 327184e39f00 registry:5000/rhosp12/openstack-swift-container-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_container_replicator a6fb84983ec9 registry:5000/rhosp12/openstack-swift-account-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_account_auditor 165c78a544d0 registry:5000/rhosp12/openstack-swift-object-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_object_replicator b3f5ab061149 registry:5000/rhosp12/openstack-swift-container-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_container_server a8cbfe2198b1 registry:5000/rhosp12/openstack-swift-object-docker:20171103.1 "kolla_start" 42 minutes ago Up 42 minutes (healthy) swift_rsync 39607802e755 registry:5000/rhosp12/openstack-swift-account-docker:20171103.1 "kolla_start" 43 minutes ago Up 42 minutes (healthy) swift_account_reaper 26995d9639fd registry:5000/rhosp12/openstack-swift-account-docker:20171103.1 "kolla_start" 43 minutes ago Up 43 minutes (healthy) swift_account_replicator 1292e6a6809b registry:5000/rhosp12/openstack-swift-object-docker:20171103.1 "kolla_start" 43 minutes ago Up 43 minutes (healthy) swift_object_auditor 6a0761ce34a9 registry:5000/rhosp12/openstack-swift-proxy-server-docker:20171103.1 "kolla_start" 43 minutes ago Up 43 minutes (healthy) swift_proxy b2c8b5b58407 registry:5000/rhosp12/openstack-swift-object-docker:20171103.1 "kolla_start" 43 minutes ago Up 43 minutes (healthy) swift_object_server 9a0ac56ff65b registry:5000/rhosp12/openstack-swift-container-docker:20171103.1 "kolla_start" 43 minutes ago Up 43 minutes (healthy) swift_container_updater
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462