Red Hat Bugzilla – Bug 1585678
[UPDATE] [SWIFT] after update openstack-swift-object stuck in restarting
Last modified: 2018-08-29 12:37:34 EDT
Description of problem: After overcloud update the openstack-swift-object get stuck in restarting state b3ec4546e62b 192.168.24.1:8787/rhosp13/openstack-swift-object:2018-05-29.2 "kolla_start" 24 hours ago Restarting (11) About an hour ago swift_rsync full output of docker ps is attached Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1447408 [details] docker ps
Can you share logfiles with us (from /var/log/containers/swift and /var/log/swift) please? Or even grant access to a deployment to investigate this? Thx!
Checked this, it's the same reason as in https://bugzilla.redhat.com/show_bug.cgi?id=1517548. $ sudo docker logs swift_rsync exec /usr/bin/rsync --daemon --no-detach --config=/etc/rsyncd.conf failed to create pid file /var/run/rsyncd.pid: File exists Question is why this happens with the fix already included. Did the update finish successfully?
So this one is a bit tricky, it really depends on the deployment - I guess one mostly sees this in slow environments? The problem is that there is a default timeout of 10 seconds in docker to stop a container. After that the process in the container will be killed. In the case of rsyncd it might get killed therefore, shutting down unclean and leaving the rsyncd.pid there. When the container is restarted, the rsyncd process finds the pid file, and immediately exits. Actually the pid file is not really used in a containerized environment, so we should get rid of it. However, this one is hardcoded into a template [1] and thus can't be simply unset in TripleO. I think we should do the following: add a workaround in puppet-tripleo or tripleo-heat-templates to comment the pid file setting, while at the same time proposing an update to puppetlabs-rsync, making the pid file configurable. Once this is merged, we can remove the workaround. The manual workaround to get the container out of this loop is to modify the pid file setting like this: sed -i -e 's/^pid file/# pid file/g' /var/lib/config-data/swift/etc/rsyncd.conf [1] https://github.com/puppetlabs/puppetlabs-rsync/blob/master/templates/header.erb#L4
Also proposed required PR for puppetlabs-rsync: https://github.com/puppetlabs/puppetlabs-rsync/pull/120
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574