Bug 1474866 - [osp12]swift_object_expirer, swift_container_replicator, swift_object_replicator, swift_proxy, wift_account_replicator node have restarting state due to "IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'
Summary: [osp12]swift_object_expirer, swift_container_replicator, swift_object_replica...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-containers
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 12.0 (Pike)
Assignee: Christian Schwede (cschwede)
QA Contact: Omri Hochman
Andrew Burden
URL:
Whiteboard:
: 1478014 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-25 14:07 UTC by Artem Hrechanychenko
Modified: 2017-12-14 14:51 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-14 14:51:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1710952 0 None None None 2017-08-26 08:46:26 UTC
OpenStack gerrit 494008 0 None None None 2017-08-26 08:45:57 UTC

Description Artem Hrechanychenko 2017-07-25 14:07:58 UTC
Description of problem:

puddle version: 2017-07-22.1 

[heat-admin@overcloud-controller-0 ~]$ sudo docker ps |grep "Restarting"
e2c6c39e2966        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1          "kolla_start"            25 minutes ago      Restarting (1) 11 minutes ago                       swift_object_expirer
64c262679038        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-container-docker:2017-07-22.1             "kolla_start"            25 minutes ago      Restarting (1) 11 minutes ago                       swift_container_replicator
4df3772b440c        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-account-docker:2017-07-22.1               "kolla_start"            25 minutes ago      Restarting (1) 11 minutes ago                       swift_account_replicator
cc5ba206968d        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-proxy-server-docker:2017-07-22.1          "kolla_start"            25 minutes ago      Restarting (1) 11 minutes ago                       swift_proxy
4430df861ba4        docker-registry.engineering.redhat.com/rhosp12/openstack-swift-object-docker:2017-07-22.1                "kolla_start"            25 minutes ago      Restarting (1) 11 minutes ago                       swift_object_replicator

Running command: '/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf'
Traceback (most recent call last):
  File "/usr/bin/swift-proxy-server", line 23, in <module>
    sys.exit(run_wsgi(conf_file, 'proxy-server', **options))
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 899, in run_wsgi
    loadapp(conf_path, global_conf=global_conf)
  File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 393, in loadapp
    app = ctx.app_context.create()
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 715, in app_factory
    app = Application(conf)
  File "/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 198, in __init__
    ring_name='container')
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 196, in __init__
    self._reload(force=True)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 201, in _reload
    ring_data = RingData.load(self.serialized_path)
  File "/usr/lib/python2.7/site-packages/swift/common/ring/ring.py", line 96, in load
    gz_file = GzipFile(filename, 'rb')
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '/etc/swift/container.ring.gz'

[heat-admin@overcloud-controller-0 ~]$ ls /var/lib/config-data/swift/etc/swift/
account.builder  account-server       backups            container-reconciler.conf  container-server       object.builder       object.ring.gz  object-server.conf  proxy-server.conf  swift.conf
account.ring.gz  account-server.conf  container.builder  container.ring.gz          container-server.conf  object-expirer.conf  object-server   proxy-server        rootwrap.conf


openstack-neutron-common-11.0.0-0.20170719132730.1c94a80.el7ost.noarch
openstack-glance-15.0.0-0.20170718113127.20ea7ab.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170719132730.1c94a80.el7ost.noarch
openstack-ironic-api-8.0.1-0.20170719072039.d9983f1.el7ost.noarch
puppet-openstack_extras-11.2.0-0.20170704143612.8932465.el7ost.noarch
openstack-nova-conductor-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-mistral-engine-5.0.0-0.20170718095321.61231ec.el7ost.noarch
openstack-heat-engine-9.0.0-0.20170719132024.923d018.el7ost.noarch
python-openstackclient-3.11.0-0.20170613232431.c69304e.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170718190543.el7ost.noarch
openstack-tripleo-common-containers-7.3.1-0.20170718114623.1d79e16.el7ost.noarch
openstack-ironic-inspector-5.1.1-0.20170705203602.c38596e.el7ost.noarch
puppet-openstacklib-11.2.0-0.20170714191355.76de885.el7ost.noarch
openstack-nova-common-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-nova-scheduler-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-mistral-api-5.0.0-0.20170718095321.61231ec.el7ost.noarch
openstack-tempest-16.1.1-0.20170719134023.2a0e141.el7ost.noarch
openstack-heat-api-cfn-9.0.0-0.20170719132024.923d018.el7ost.noarch
openstack-swift-container-2.14.1-0.20170718054917.3c11f6b.el7ost.noarch
openstack-tripleo-validations-7.1.1-0.20170717141229.ce35d5f.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
python-openstacksdk-0.9.17-0.20170621195806.7946243.el7ost.noarch
openstack-heat-common-9.0.0-0.20170719132024.923d018.el7ost.noarch
openstack-mistral-common-5.0.0-0.20170718095321.61231ec.el7ost.noarch
openstack-nova-placement-api-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-nova-api-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170719132730.1c94a80.el7ost.noarch
openstack-keystone-12.0.0-0.20170718172821.239bc36.el7ost.noarch
openstack-heat-api-9.0.0-0.20170719132024.923d018.el7ost.noarch
openstack-tripleo-image-elements-7.0.0-0.20170712081605.35068ac.el7ost.noarch
openstack-swift-account-2.14.1-0.20170718054917.3c11f6b.el7ost.noarch
python-openstack-mistral-5.0.0-0.20170718095321.61231ec.el7ost.noarch
openstack-nova-compute-16.0.0-0.20170719155122.7ae3753.el7ost.noarch
openstack-mistral-executor-5.0.0-0.20170718095321.61231ec.el7ost.noarch
openstack-zaqar-5.0.0-0.20170719124338.13b85cc.el7ost.noarch
openstack-selinux-0.8.8-0.20170622195307.74ddc0e.el7ost.noarch
openstack-swift-proxy-2.14.1-0.20170718054917.3c11f6b.el7ost.noarch
openstack-tripleo-ui-7.1.1-0.20170718122426.8337319.el7ost.noarch
openstack-tripleo-common-7.3.1-0.20170718114623.1d79e16.el7ost.noarch
openstack-ironic-common-8.0.1-0.20170719072039.d9983f1.el7ost.noarch
openstack-neutron-11.0.0-0.20170719132730.1c94a80.el7ost.noarch
openstack-ironic-conductor-8.0.1-0.20170719072039.d9983f1.el7ost.noarch
openstack-swift-object-2.14.1-0.20170718054917.3c11f6b.el7ost.noarch
openstack-tripleo-puppet-elements-7.0.0-0.20170715003644.4092ef5.el7ost.noarch

Comment 7 Artem Hrechanychenko 2017-08-04 09:00:18 UTC
There a few bugs with same symptoms
https://bugzilla.redhat.com/show_bug.cgi?id=1475379
https://bugzilla.redhat.com/show_bug.cgi?id=1470194

Comment 10 Thiago da Silva 2017-08-10 01:39:49 UTC
*** Bug 1478014 has been marked as a duplicate of this bug. ***

Comment 11 Carlos Camacho 2017-08-15 08:25:04 UTC
Hey I havent seen this error on deployment, only when upgrading from Ocata to Pike (containerized).

In the case of the upgrade is that we are mapping "/var/lib/config-data/puppet-generated/swift/" to "/etc/swift" but we havent ran puppet to reconfigure all again and you will have this issue.

Comment 12 Martin André 2017-08-26 08:45:58 UTC
A related bugfix was submitted upstream in https://review.openstack.org/#/c/494008/.

Comment 13 Christian Schwede (cschwede) 2017-08-31 12:52:37 UTC
Moving to ON_DEV, upstream fix merged: https://review.openstack.org/#/c/494008/


Note You need to log in before you can comment on or make changes to this bug.