Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1900015

Summary: podman container storage might get out of sync after a host crash
Product: Red Hat Enterprise Linux 8 Reporter: Damien Ciabrini <dciabrin>
Component: resource-agentsAssignee: dabarzil
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.2CC: agk, cfeist, cluster-maint, dabarzil, fdinitto, mjuricek, pkomarov
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: 8.0Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: resource-agents-4.1.1-76.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1975903 1975904 (view as bug list) Environment:
Last Closed: 2021-05-18 15:11:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1975903, 1975904    

Description Damien Ciabrini 2020-11-20 15:31:04 UTC
Description of problem:

I've hit on a HA overcloud while doing destructive testing:

  . when simulating a network partition, a pacemaker host notices that it
    no longer has quorum,

  . this pacemaker host stops all the podman containers it's managing
    locally (e.g. local instances of haproxy, galera, rabbit...)

  . the other pacemaker peers try to fence it, because they also noticed
    the one node was partitionned.

  . the fencing of the partitionned node takes place successfully, while
    some podman containers are still being stopped on that node.

When the host is fenced while podman is stopping a container, it might
happen that a regular 'podman rm' operation doesn't finish entirely;
part of the container information has been removed from podman, while
another part it is still present in podman's container storage.

When that happens, the podman CLI tells us that container foo doesn't
exist anymore:

    podman inspect rabbitmq-bundle-podman-1
    Error: error getting image "rabbitmq-bundle-podman-1": unable to find a name and tag match for rabbitmq-bundle-podman-1 in repotags: no such image

But when trying to create a new container with the same name, podman will
fail with an internal error:

    Nov 06 21:40:06 standalone-1 podman(rabbitmq-bundle-podman-1)[444865]: ERROR: Error: error creating container storage: the container name "rabbitmq-bundle-podman-1" is already in use by "2725a8856261fde79b26b680e18b5ae95a797801fd9e2ab0c379a5ee6a1a3ee6". You have to remove >
    Nov 06 21:40:06 standalone-1 podman(rabbitmq-bundle-podman-1)[444872]: ERROR: podman failed to launch container

When that happens, the podman resource-agent is stuck and cannot
recreate the necessary container on the host. The only way out is to
manually force the deletion of the resource with:

    podman rm --storage rabbitmq-bundle-podman-1

Otherwise, the resource will get stuck forever on the host and the
service will be impacted globally in the OpenStack control plane.


Version-Release number of selected component (if applicable):
resource-agents-4.1.1-44.el8_2.6.x86_64

How reproducible:
Random

Steps to Reproduce:
1. Deploy an overcloud
2. trigger a network split to isolate one node from the cluster
3. once the node is fenced and comes back up, remove the network split

Actual results:
Sometimes, some containers can't be recreated, and the service is impacted

Expected results:
containers should always be recreated

Additional info:

Comment 4 dabarzil 2021-01-13 15:27:03 UTC
There's my reproduce:

[stack@undercloud-0 ~]$ ansible controller -b -mshell -a'podman exec `podman ps -f name=galera-bundle -q`  sh -c "rpm -q resource-agents";rpm -q resource-agents'
[WARNING]: Found both group and host with same name: undercloud
controller-1 | CHANGED | rc=0 >>
resource-agents-4.1.1-76.el8.x86_64
resource-agents-4.1.1-76.el8.x86_64
controller-0 | CHANGED | rc=0 >>
resource-agents-4.1.1-76.el8.x86_64
resource-agents-4.1.1-76.el8.x86_64
controller-2 | CHANGED | rc=0 >>
resource-agents-4.1.1-76.el8.x86_64
resource-agents-4.1.1-76.el8.x86_64

[stack@undercloud-0 ~]$ ansible controller -b -mshell -a'podman commit `podman ps -f name=rabbitmq-bundle -q` cluster.common.tag/rhosp16-openstack-rabbitmq:pcmklatest'
[WARNING]: Found both group and host with same name: undercloud
controller-0 | CHANGED | rc=0 >>
6fac9a35f08b60c298fae686e0b19e097fe52391dd468763d88a2ee29b423141time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/attrd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/cib_ro to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/cib_rw to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/cib_shm to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/crmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/lrmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/6c9ad7e8fd3b860b82c056cef2385a54dc0edf4dd73e1f99e0d8e4e4b0854a95/diff/run/stonith-ng to tar: archive/tar: sockets not supported"
Getting image source signatures
Copying blob sha256:086fdf710ad59645f326763e3e0d7801be9741c88b652dd1875128d0dd98c54b
Copying blob sha256:6388cbf42149eb02ceebcfac6a9b102d47a08889f55fc4d15ceb1ea69205eeb0
Copying blob sha256:a95305b0d5a2cdea24b8f55dca3449240f4376b2595da032b52b918469dd8d10
Copying blob sha256:34bf9afbf2c07091a9de0af54ed4c4e88d3884b9a447fe59b2207644b6b3be85
Copying blob sha256:ea805c919b34a402ce0bce12f3f9eda8690bfb85d903d81536d23545c3529c9d
Copying config sha256:6fac9a35f08b60c298fae686e0b19e097fe52391dd468763d88a2ee29b423141
Writing manifest to image destination
Storing signatures
controller-1 | CHANGED | rc=0 >>
ca003d6e8aa5d1dcc068021467d13f756c58cd35394d8dac20c921b9e56c0cdbtime="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/attrd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/cib_ro to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/cib_rw to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/cib_shm to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/crmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/lrmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/a9ed1702ba37b01550f07d034f231e056e9ece7e26fdbfe240b5017ddc19ed9e/diff/run/stonith-ng to tar: archive/tar: sockets not supported"
Getting image source signatures
Copying blob sha256:086fdf710ad59645f326763e3e0d7801be9741c88b652dd1875128d0dd98c54b
Copying blob sha256:6388cbf42149eb02ceebcfac6a9b102d47a08889f55fc4d15ceb1ea69205eeb0
Copying blob sha256:a95305b0d5a2cdea24b8f55dca3449240f4376b2595da032b52b918469dd8d10
Copying blob sha256:34bf9afbf2c07091a9de0af54ed4c4e88d3884b9a447fe59b2207644b6b3be85
Copying blob sha256:6281e590bff2ae86d0f2740873ec6ec480259f1380ef4b1edb991a0dae6bdc92
Copying config sha256:ca003d6e8aa5d1dcc068021467d13f756c58cd35394d8dac20c921b9e56c0cdb
Writing manifest to image destination
Storing signatures
controller-2 | CHANGED | rc=0 >>
0dfdfcf6e441ec170916eaab732ef05da60cd8c40b2dd58bf256b89c7164d9f0time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/attrd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/cib_ro to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/cib_rw to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/cib_shm to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/crmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/lrmd to tar: archive/tar: sockets not supported"
time="2021-01-13T12:20:16Z" level=error msg="Can't add file /var/lib/containers/storage/overlay/e75c3a0e7b3acf41ea2862033a250e1072c16c706bcbd6fa8da967733011c8a5/diff/run/stonith-ng to tar: archive/tar: sockets not supported"
Getting image source signatures
Copying blob sha256:086fdf710ad59645f326763e3e0d7801be9741c88b652dd1875128d0dd98c54b
Copying blob sha256:6388cbf42149eb02ceebcfac6a9b102d47a08889f55fc4d15ceb1ea69205eeb0
Copying blob sha256:a95305b0d5a2cdea24b8f55dca3449240f4376b2595da032b52b918469dd8d10
Copying blob sha256:34bf9afbf2c07091a9de0af54ed4c4e88d3884b9a447fe59b2207644b6b3be85
Copying blob sha256:3d53be386afc7235e0895d86aefe0f7a7e04ebca7ef27856891c5e492d9618f7
Copying config sha256:0dfdfcf6e441ec170916eaab732ef05da60cd8c40b2dd58bf256b89c7164d9f0
Writing manifest to image destination
Storing signatures

[root@controller-0 ~]# ifdown br-isolated
WARN      : [ifdown] You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
WARN      : [ifdown] 'network-scripts' will be removed in one of the next major releases of RHEL.
WARN      : [ifdown] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
[root@controller-0 ~]# 
Broadcast message from systemd-journald@controller-0 (Wed 2021-01-13 12:25:27 UTC):

haproxy[259975]: proxy mysql has no server available!

client_loop: send disconnect: Broken pipe

[heat-admin@controller-1 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Cluster Summary:
  * Stack: corosync
  * Current DC: controller-2 (version 2.0.4-6.el8-2deceaa3ae) - partition with quorum
  * Last updated: Wed Jan 13 15:21:51 2021
  * Last change:  Wed Jan 13 12:25:35 2021 by redis-bundle-1 via crm_attribute on controller-2
  * 15 nodes configured
  * 50 resource instances configured

Node List:
  * Online: [ controller-0 controller-1 controller-2 ]
  * GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-2 galera-bundle-2@controller-1 ovn-dbs-bundle-0@controller-2 ovn-dbs-bundle-1@controller-0 ovn-dbs-bundle-2@controller-1 rabbitmq-bundle-0@controller-2 rabbitmq-bundle-1@controller-0 rabbitmq-bundle-2@controller-1 redis-bundle-0@controller-0 redis-bundle-1@controller-2 redis-bundle-2@controller-1 ]

Full List of Resources:
  * Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]:
    * galera-bundle-0	(ocf::heartbeat:galera):	 Master controller-0
    * galera-bundle-1	(ocf::heartbeat:galera):	 Master controller-2
    * galera-bundle-2	(ocf::heartbeat:galera):	 Master controller-1
  * Container bundle set: rabbitmq-bundle [cluster.common.tag/rhosp16-openstack-rabbitmq:pcmklatest]:
    * rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	 Started controller-2
    * rabbitmq-bundle-1	(ocf::heartbeat:rabbitmq-cluster):	 Started controller-0
    * rabbitmq-bundle-2	(ocf::heartbeat:rabbitmq-cluster):	 Started controller-1
  * Container bundle set: redis-bundle [cluster.common.tag/rhosp16-openstack-redis:pcmklatest]:
    * redis-bundle-0	(ocf::heartbeat:redis):	 Slave controller-0
    * redis-bundle-1	(ocf::heartbeat:redis):	 Master controller-2
    * redis-bundle-2	(ocf::heartbeat:redis):	 Slave controller-1
  * ip-192.168.24.49	(ocf::heartbeat:IPaddr2):	 Started controller-2
  * ip-2620.52.0.13b8.5054.ff.fe3e.5a	(ocf::heartbeat:IPaddr2):	 Started controller-1
  * ip-fd00.fd00.fd00.2000..375	(ocf::heartbeat:IPaddr2):	 Started controller-2
  * ip-fd00.fd00.fd00.2000..1ce	(ocf::heartbeat:IPaddr2):	 Started controller-1
  * ip-fd00.fd00.fd00.3000..186	(ocf::heartbeat:IPaddr2):	 Started controller-1
  * ip-fd00.fd00.fd00.4000..2a1	(ocf::heartbeat:IPaddr2):	 Started controller-2
  * Container bundle set: haproxy-bundle [cluster.common.tag/rhosp16-openstack-haproxy:pcmklatest]:
    * haproxy-bundle-podman-0	(ocf::heartbeat:podman):	 Started controller-0
    * haproxy-bundle-podman-1	(ocf::heartbeat:podman):	 Started controller-1
    * haproxy-bundle-podman-2	(ocf::heartbeat:podman):	 Started controller-2
  * Container bundle set: ovn-dbs-bundle [cluster.common.tag/rhosp16-openstack-ovn-northd:pcmklatest]:
    * ovn-dbs-bundle-0	(ocf::ovn:ovndb-servers):	 Master controller-2
    * ovn-dbs-bundle-1	(ocf::ovn:ovndb-servers):	 Slave controller-0
    * ovn-dbs-bundle-2	(ocf::ovn:ovndb-servers):	 Slave controller-1
  * ip-fd00.fd00.fd00.2000..2a2	(ocf::heartbeat:IPaddr2):	 Started controller-2
  * stonith-fence_ipmilan-52540073a56a	(stonith:fence_ipmilan):	 Started controller-1
  * stonith-fence_ipmilan-52540097b626	(stonith:fence_ipmilan):	 Started controller-2
  * stonith-fence_ipmilan-52540009e3b2	(stonith:fence_ipmilan):	 Started controller-2
  * Container bundle: openstack-cinder-volume [cluster.common.tag/rhosp16-openstack-cinder-volume:pcmklatest]:
    * openstack-cinder-volume-podman-0	(ocf::heartbeat:podman):	 Started controller-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Comment 6 errata-xmlrpc 2021-05-18 15:11:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (resource-agents bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1736