Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1475379 - OSP11 -> OSP12 upgrade: swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_rsync, swift_account_replicator, swift_proxy containers are restarting after upgrade
OSP11 -> OSP12 upgrade: swift_object_expirer, swift_container_replicator, swi...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
high Severity urgent
: beta
: 12.0 (Pike)
Assigned To: Jose Luis Franco
Marius Cornea
: Triaged
Depends On:
Blocks: 1399762
  Show dependency tree / graph
 
Reported: 2017-07-26 10:21 EDT by Marius Cornea
Modified: 2018-02-05 14:10 EST (History)
12 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.1-0.20170927205938.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-12-13 16:44:48 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1710606 None None None 2017-08-14 07:57 EDT
OpenStack gerrit 494008 None None None 2017-08-24 10:42 EDT
OpenStack gerrit 499457 None None None 2017-10-10 09:55 EDT
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-15 20:43:25 EST

  None (edit)
Description Marius Cornea 2017-07-26 10:21:44 EDT 
Description of problem:
OSP11 -> OSP12 upgrade: swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_rsync, swift_account_replicator, swift_proxy containers are restarting after upgrade:

[root@controller-0 heat-admin]# docker ps | grep swift | grep Restart
72d5388ac7b0        192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1          "kolla_start"            About an hour ago   Restarting (1) 52 minutes ago                        swift_object_expirer
ce808ac19a31        192.168.24.1:8787/rhosp12/openstack-swift-container-docker:2017-07-22.1             "kolla_start"            About an hour ago   Restarting (1) 52 minutes ago                        swift_container_replicator
8e2824bb8aa1        192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-07-22.1                "kolla_start"            About an hour ago   Restarting (1) 52 minutes ago                        swift_object_replicator
f46bfd80ca5c        192.168.24.1:8787/rhosp12/openstack-swift-object-docker:2017-07-22.1                "kolla_start"            About an hour ago   Restarting (10) 52 minutes ago                       swift_rsync
6706b3a3cd84        192.168.24.1:8787/rhosp12/openstack-swift-account-docker:2017-07-22.1               "kolla_start"            About an hour ago   Restarting (1) 52 minutes ago                        swift_account_replicator
251154c79f13        192.168.24.1:8787/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1          "kolla_start"            About an hour ago   Restarting (1) 52 minutes ago                        swift_proxy


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170718190543.el7ost.noarch

How reproducible:
100%


Steps to Reproduce:
1. Deploy OSP11

2. Upgrade to OSP12:
#!/bin/bash

timeout 100m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \

Actual results:
swift containers are in Restarting status.

Expected results:
swift containers are up

Additional info:

[root@controller-0 heat-admin]# docker logs --tail 10 swift_proxy 
    ring_name='container')
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__
    self._reload(force=True)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload
    ring_data = RingData.load(self.serialized_path)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load
    gz_file = GzipFile(filename, 'rb')
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'
[root@controller-0 heat-admin]# docker logs --tail 10 swift_object_expirer
    ring_name='container')
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__
    self._reload(force=True)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload
    ring_data = RingData.load(self.serialized_path)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load
    gz_file = GzipFile(filename, 'rb')
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'
[root@controller-0 heat-admin]# docker logs --tail 10 swift_container_replicator
    self.ring = ring.Ring(swift_dir, ring_name=self.server_type)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__
    self._reload(force=True)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload
    ring_data = RingData.load(self.serialized_path)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load
    gz_file = GzipFile(filename, 'rb')
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'
[root@controller-0 heat-admin]# docker logs --tail 10 swift_object_replicator
INFO:__main__:Deleting /etc/swift/swift.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf
INFO:__main__:Deleting /etc/xinetd.d/rsync
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.d/rsync to /etc/xinetd.d/rsync
INFO:__main__:Deleting /etc/rsyncd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf
INFO:__main__:Deleting /etc/xinetd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.conf to /etc/xinetd.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/bin/swift-object-replicator /etc/swift/object-server.conf'
[root@controller-0 heat-admin]# docker logs --tail 10 swift_rsync
INFO:__main__:Deleting /etc/swift/swift.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/swift/swift.conf to /etc/swift/swift.conf
INFO:__main__:Deleting /etc/xinetd.d/rsync
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.d/rsync to /etc/xinetd.d/rsync
INFO:__main__:Deleting /etc/rsyncd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/rsyncd.conf to /etc/rsyncd.conf
INFO:__main__:Deleting /etc/xinetd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/xinetd.conf to /etc/xinetd.conf
INFO:__main__:Writing out command to execute
Running command: '/usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf'
[root@controller-0 heat-admin]# docker logs --tail 10 swift_account_replicator
    self.ring = ring.Ring(swift_dir, ring_name=self.server_type)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__
    self._reload(force=True)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload
    ring_data = RingData.load(self.serialized_path)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load
    gz_file = GzipFile(filename, 'rb')
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/etc/swift/account.ring.gz'
Comment 1 Amit Ugol 2017-07-30 06:13:59 EDT 
The missing file is there during 11, so my guess here is that during the migration of concurrent services into containers (and therefore cleaning them from the node) the file is removed which causes the IO error.
Temp assigning to Marios for our next triaging meeting tomorrow.
Comment 2 Marius Cornea 2017-07-31 04:23:32 EDT 
There is a pretty much similar bug reported for osp12 fresh deployment: bug 1474866 so this might be a duplicate.
Comment 3 Marius Cornea 2017-08-01 04:19:48 EDT 
There are no ring files in /var/lib/config-data/puppet-generated/swift/etc/swift/

[root@controller-0 heat-admin]# ls -l /var/lib/config-data/puppet-generated/swift/etc/swift/
total 28
drwxr-xr-x. 2 42445 42445    6 Jul 31 17:34 account-server
-rw-r--r--. 1 42445 42445  656 Jul 31 17:34 account-server.conf
drwxr-xr-x. 2 42445 42445    6 Jul 31 17:34 container-server
-rw-r--r--. 1 42445 42445  745 Jul 31 17:34 container-server.conf
-rw-r-----. 1 root  42445  527 Jul 31 17:34 object-expirer.conf
drwxr-xr-x. 2 42445 42445    6 Jul 31 17:34 object-server
-rw-r--r--. 1 42445 42445  679 Jul 31 17:34 object-server.conf
-rw-r-----. 1 root  42445 4211 Jul 31 17:34 proxy-server.conf
-rw-r-----. 1 42445 42445  106 Jul 31 17:34 swift.conf

but these are present in /etc/swift:

[root@controller-0 heat-admin]# ls -l /etc/swift/
total 84
-rw-r--r--. 1 swift swift 9277 Jul 31 15:18 account.builder
-rw-r--r--. 1 root  root   249 Jul 31 15:18 account.ring.gz
drwxr-xr-x. 2 root  root     6 Jul 26 07:01 account-server
-rw-r--r--. 1 swift swift  656 Jul 31 15:28 account-server.conf
drwxr-xr-x. 2 swift swift 4096 Jul 31 15:18 backups
-rw-r--r--. 1 swift swift 9277 Jul 31 15:18 container.builder
-rw-r-----. 1 root  swift 1415 Jul 26 06:53 container-reconciler.conf
-rw-r--r--. 1 root  root   250 Jul 31 15:18 container.ring.gz
drwxr-xr-x. 2 root  root     6 Jul 26 07:01 container-server
-rw-r--r--. 1 swift swift  745 Jul 31 15:28 container-server.conf
-rw-r--r--. 1 swift swift 9277 Jul 31 15:18 object.builder
-rw-r-----. 1 root  swift  529 Jul 31 15:24 object-expirer.conf
-rw-r--r--. 1 root  root   247 Jul 31 15:18 object.ring.gz
drwxr-xr-x. 2 root  root     6 Jul 26 07:01 object-server
-rw-r--r--. 1 swift swift  655 Jul 31 15:28 object-server.conf
drwxr-xr-x. 2 root  root     6 Jul 26 07:01 proxy-server
-rw-r-----. 1 root  swift 4809 Jul 31 15:28 proxy-server.conf
-rw-r-----. 1 swift swift  106 Jul 31 15:14 swift.conf
Comment 4 Marius Cornea 2017-08-01 04:20:52 EDT 
After running:

cp /etc/swift/*gz /var/lib/config-data/puppet-generated/swift/etc/swift/
for container in swift_proxy swift_object_expirer swift_container_replicator swift_object_replicator swift_rsync swift_account_replicator; do docker stop $container; docker start $container; done

only the swift_rsync container keeps restarting.
Comment 5 Jose Luis Franco 2017-08-11 04:04:57 EDT 
It seems that some manual steps are needed to configure swift: 
http://tripleo.org/install/advanced_deployment/deploy_swift.html

Which in fact, it is related to the workaround Marius posted, copy the files manually and restart the service. But I wonder, couldn't this be automated?
Comment 6 Jon Schlueter 2017-10-10 09:55:25 EDT 
updated to include stable/pike review upstream
Comment 9 Marius Cornea 2017-11-08 11:22:46 EST 
[root@controller-0 heat-admin]# docker ps | grep swift 

213996a6ca16        registry:5000/rhosp12/openstack-swift-account-docker:20171103.1             "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_account_server
2cbb553fcc67        registry:5000/rhosp12/openstack-swift-container-docker:20171103.1           "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_container_auditor
8eb669ac1b56        registry:5000/rhosp12/openstack-swift-proxy-server-docker:20171103.1        "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_object_expirer
a74274d06532        registry:5000/rhosp12/openstack-swift-object-docker:20171103.1              "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_object_updater
327184e39f00        registry:5000/rhosp12/openstack-swift-container-docker:20171103.1           "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_container_replicator
a6fb84983ec9        registry:5000/rhosp12/openstack-swift-account-docker:20171103.1             "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_account_auditor
165c78a544d0        registry:5000/rhosp12/openstack-swift-object-docker:20171103.1              "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_object_replicator
b3f5ab061149        registry:5000/rhosp12/openstack-swift-container-docker:20171103.1           "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_container_server
a8cbfe2198b1        registry:5000/rhosp12/openstack-swift-object-docker:20171103.1              "kolla_start"            42 minutes ago      Up 42 minutes (healthy)                       swift_rsync
39607802e755        registry:5000/rhosp12/openstack-swift-account-docker:20171103.1             "kolla_start"            43 minutes ago      Up 42 minutes (healthy)                       swift_account_reaper
26995d9639fd        registry:5000/rhosp12/openstack-swift-account-docker:20171103.1             "kolla_start"            43 minutes ago      Up 43 minutes (healthy)                       swift_account_replicator
1292e6a6809b        registry:5000/rhosp12/openstack-swift-object-docker:20171103.1              "kolla_start"            43 minutes ago      Up 43 minutes (healthy)                       swift_object_auditor
6a0761ce34a9        registry:5000/rhosp12/openstack-swift-proxy-server-docker:20171103.1        "kolla_start"            43 minutes ago      Up 43 minutes (healthy)                       swift_proxy
b2c8b5b58407        registry:5000/rhosp12/openstack-swift-object-docker:20171103.1              "kolla_start"            43 minutes ago      Up 43 minutes (healthy)                       swift_object_server
9a0ac56ff65b        registry:5000/rhosp12/openstack-swift-container-docker:20171103.1           "kolla_start"            43 minutes ago      Up 43 minutes (healthy)                       swift_container_updater
Comment 13 errata-xmlrpc 2017-12-13 16:44:48 EST 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.