Bug 1746113
| Summary: | swift containers are restarted at every redeploy with no changes | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Michele Baldessari <michele> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Christian Schwede (cschwede) <cschwede> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Joe H. Rahme <jhakimra> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 16.0 (Train) | CC: | bdobreli, mburns, stchen, tkajinam |
| Target Milestone: | --- | Keywords: | Reopened, Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | Triaged | ||
| Fixed In Version: | openstack-tripleo-heat-templates-11.3.2-1.20210408163453.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-12-02 12:59:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 1692217 has been marked as a duplicate of this bug. *** Closing EOL, OSP 15 has been retired as of Sept 19, 2020 |
Description of problem: This has been observed on a master deployment but it seems osp15 is also affected. Just redeploy an overcloud with no changes whatsoever (make sure you wait some time between the initial deploy and the subsequent one) and you will observe most (all?) swift container having been restarted: 130 ↵ 360c61babec2 192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes (healthy) swift_proxy db0b91efbc18 192.168.24.1:8787/tripleomaster/centos-binary-swift-container:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_container_auditor 0f213241fa09 192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_object_expirer ebc3946c9211 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_object_updater 63dcc733696e 192.168.24.1:8787/tripleomaster/centos-binary-swift-container:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_container_replicator acb24e7570ea 192.168.24.1:8787/tripleomaster/centos-binary-swift-account:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_account_auditor 415e8d2a4b5e 192.168.24.1:8787/tripleomaster/centos-binary-swift-account:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes (healthy) swift_account_server 66a2af2f4b71 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_object_replicator 4d0176eb79d0 192.168.24.1:8787/tripleomaster/centos-binary-swift-container:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes (healthy) swift_container_server 10a46d69f4fc 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_rsync 8ad77e2d9faa 192.168.24.1:8787/tripleomaster/centos-binary-swift-account:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_account_reaper fe963362a1bc 192.168.24.1:8787/tripleomaster/centos-binary-swift-account:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_account_replicator d746d52d1c31 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_object_auditor d4ec3dde9c22 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes (healthy) swift_object_server aba4af798da9 192.168.24.1:8787/tripleomaster/centos-binary-swift-container:current-tripleo-rdo "dumb-init --singl..." 14 minutes ago Up 14 minutes swift_container_updater a2f523858cc7 192.168.24.1:8787/tripleomaster/centos-binary-swift-object:current-tripleo-rdo "dumb-init --singl..." 21 minutes ago Exited (0) 21 minutes ago swift_rsync_fix I put /var/lib/config/puppet-generated under git control and it seems that the following files trigger the restart: diff --git a/swift_ringbuilder/etc/swift/account.builder b/swift_ringbuilder/etc/swift/account.builder index 94f1c7e..f0cb3d5 100644 Binary files a/swift_ringbuilder/etc/swift/account.builder and b/swift_ringbuilder/etc/swift/account.builder differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849266.object.builder b/swift_ringbuilder/etc/swift/backups/1566849266.object.builder deleted file mode 100644 index cb086a3..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849266.object.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849267.account.builder b/swift_ringbuilder/etc/swift/backups/1566849267.account.builder deleted file mode 100644 index 9e66fe4..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849267.account.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849269.container.builder b/swift_ringbuilder/etc/swift/backups/1566849269.container.builder deleted file mode 100644 index aea46c6..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849269.container.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849291.object.builder b/swift_ringbuilder/etc/swift/backups/1566849291.object.builder deleted file mode 100644 index 129c520..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849291.object.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849291.object.ring.gz b/swift_ringbuilder/etc/swift/backups/1566849291.object.ring.gz deleted file mode 100644 index 475e555..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849291.object.ring.gz and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849292.account.builder b/swift_ringbuilder/etc/swift/backups/1566849292.account.builder deleted file mode 100644 index 94f1c7e..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849292.account.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849292.account.ring.gz b/swift_ringbuilder/etc/swift/backups/1566849292.account.ring.gz deleted file mode 100644 index e5436c8..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849292.account.ring.gz and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849293.container.builder b/swift_ringbuilder/etc/swift/backups/1566849293.container.builder deleted file mode 100644 index c3d24e2..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849293.container.builder and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/backups/1566849293.container.ring.gz b/swift_ringbuilder/etc/swift/backups/1566849293.container.ring.gz deleted file mode 100644 index dfaaaba..0000000 Binary files a/swift_ringbuilder/etc/swift/backups/1566849293.container.ring.gz and /dev/null differ diff --git a/swift_ringbuilder/etc/swift/container.builder b/swift_ringbuilder/etc/swift/container.builder index c3d24e2..e11cf54 100644 Binary files a/swift_ringbuilder/etc/swift/container.builder and b/swift_ringbuilder/etc/swift/container.builder differ diff --git a/swift_ringbuilder/etc/swift/object.builder b/swift_ringbuilder/etc/swift/object.builder index 129c520..af8ac2f 100644 Binary files a/swift_ringbuilder/etc/swift/object.builder and b/swift_ringbuilder/etc/swift/object.builder differ I realize that we do want to keep those files and folders in the puppet-generated folder and we actually reimposed that via: commit b49629f085b5f04be999e0d5697a77e3791f7dcc Author: Christian Schwede <cschwede> Date: Wed Nov 7 08:53:48 2018 +0100 Do not ignore Swift ring changes to trigger container restart Swift containers need to restart if the rings change. In non-containerized environments this is not required, because the Swift processes will reload the rings on any changes. However, this does not work within containers, thus a restart is required. This also restarts swift_copy_rings and swift_setup_srv container. This will copy the updated ring files and ensure new storage mount points are using the right permissions. Closes-Bug: 1802066 Related-Bug: 1786065 Change-Id: Ie2b9f003dc34f2f02a45293d06d6a40c8d5ed8ff But maybe it is time to explore alternative solutions? (Maybe not storing those builder+backups in /etc/swift?)