Bug 2123501

Summary: [RDR] Pod stuck due to error "applyFSGroup failed for vol" for a PVC that was relocated
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Shyamsundar <srangana>
Component: odf-drAssignee: Shyamsundar <srangana>
odf-dr sub component: ramen QA Contact: kmanohar
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: amagrawa, cblum, ebenahar, hekumar, idryomov, jespy, kramdoss, kseeger, mrajanna, muagarwa, ocs-bugs, odf-bz-bot, prsurve, sheggodu, srangana, ypadia
Version: 4.11   
Target Milestone: ---   
Target Release: ODF 4.12.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.12.0-74 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 2112703
: 2123811 (view as bug list) Environment:
Last Closed: 2023-03-08 15:31:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 34 kmanohar 2023-01-31 04:52:32 UTC 
verification comments:-

Pod,pvc, vr, vrg status after performing multiple relocate operations

Didn't observe any fs corruption

oc get vr,vrg,pod,pvc -n busybox-workloads-1
NAME                                                                AGE   VOLUMEREPLICATIONCLASS                 PVCNAME          DESIREDSTATE   CURRENTSTATE
volumereplication.replication.storage.openshift.io/busybox-pvc-1    39h   rbd-volumereplicationclass-473128587   busybox-pvc-1    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-10   39h   rbd-volumereplicationclass-473128587   busybox-pvc-10   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-11   39h   rbd-volumereplicationclass-473128587   busybox-pvc-11   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-12   39h   rbd-volumereplicationclass-473128587   busybox-pvc-12   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-13   39h   rbd-volumereplicationclass-473128587   busybox-pvc-13   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-14   39h   rbd-volumereplicationclass-473128587   busybox-pvc-14   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-15   39h   rbd-volumereplicationclass-473128587   busybox-pvc-15   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-16   39h   rbd-volumereplicationclass-473128587   busybox-pvc-16   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-17   39h   rbd-volumereplicationclass-473128587   busybox-pvc-17   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-18   39h   rbd-volumereplicationclass-473128587   busybox-pvc-18   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-19   39h   rbd-volumereplicationclass-473128587   busybox-pvc-19   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-2    39h   rbd-volumereplicationclass-473128587   busybox-pvc-2    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-20   39h   rbd-volumereplicationclass-473128587   busybox-pvc-20   primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-3    39h   rbd-volumereplicationclass-473128587   busybox-pvc-3    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-4    39h   rbd-volumereplicationclass-473128587   busybox-pvc-4    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-5    39h   rbd-volumereplicationclass-473128587   busybox-pvc-5    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-6    39h   rbd-volumereplicationclass-473128587   busybox-pvc-6    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-7    39h   rbd-volumereplicationclass-473128587   busybox-pvc-7    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-8    39h   rbd-volumereplicationclass-473128587   busybox-pvc-8    primary        Primary
volumereplication.replication.storage.openshift.io/busybox-pvc-9    39h   rbd-volumereplicationclass-473128587   busybox-pvc-9    primary        Primary

NAME                                                       DESIREDSTATE   CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/busybox-drpc   primary        Primary

NAME                              READY   STATUS    RESTARTS   AGE
pod/busybox-1-8cb4bd97c-jz68t     1/1     Running   0          39h
pod/busybox-10-6b797ddd86-jmzxv   1/1     Running   0          39h
pod/busybox-11-67bd665b5b-6bj85   1/1     Running   0          39h
pod/busybox-12-756c7f986-jwmlx    1/1     Running   0          39h
pod/busybox-13-767cc75996-w6gnc   1/1     Running   0          39h
pod/busybox-14-6b96d799cd-4lm5s   1/1     Running   0          39h
pod/busybox-15-54fcb85f6-nnbkz    1/1     Running   0          39h
pod/busybox-16-b45857946-nzz2v    1/1     Running   0          39h
pod/busybox-17-dd479cdc6-jrckj    1/1     Running   0          39h
pod/busybox-18-844ffc74b-nqwrt    1/1     Running   0          39h
pod/busybox-19-968d4f8c6-f8pkq    1/1     Running   0          39h
pod/busybox-2-689fdcdf54-npds2    1/1     Running   0          39h
pod/busybox-20-75fddf946d-tvv6h   1/1     Running   0          39h
pod/busybox-3-7867d488fd-q84jc    1/1     Running   0          39h
pod/busybox-4-5795ddc44b-qjtzc    1/1     Running   0          39h
pod/busybox-5-7486849478-hm28b    1/1     Running   0          39h
pod/busybox-6-69d79b6674-c8hkw    1/1     Running   0          39h
pod/busybox-7-7ddb544cdb-jcfkz    1/1     Running   0          39h
pod/busybox-8-67d78fbf78-g47xd    1/1     Running   0          39h
pod/busybox-9-7b458df98b-bpvjx    1/1     Running   0          39h

NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
persistentvolumeclaim/busybox-pvc-1    Bound    pvc-b61d6c29-9575-491f-8c1e-54501fc2fa7e   94Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-10   Bound    pvc-2e961092-1cc0-4605-b948-41894962e123   87Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-11   Bound    pvc-d91d8109-ccd1-40dd-b665-9f122c90dec0   33Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-12   Bound    pvc-cd47bfe0-e3de-43ef-b057-6868329b9226   147Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-13   Bound    pvc-1faed007-6885-46b1-b8d3-1f97d9f3317e   77Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-14   Bound    pvc-90eed2f9-9977-495c-ba42-dbd993797dc7   70Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-15   Bound    pvc-d97f7d69-6280-4468-b4c8-5d924d43cc00   131Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-16   Bound    pvc-fd698c5f-2ec8-422c-8c15-7cb8d5f527bd   127Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-17   Bound    pvc-9823e517-a3b1-4acb-b412-cc1444047d72   58Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-18   Bound    pvc-1c3e0063-cf90-4933-825e-347e570483ae   123Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-19   Bound    pvc-742b6eae-cd15-4ae5-8be1-0842801ed2f2   61Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-2    Bound    pvc-939c082b-a3be-4466-80bd-cfb9760defee   44Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-20   Bound    pvc-687c8190-8e6d-4154-a0ca-e44b456b4bff   33Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-3    Bound    pvc-dc252619-9aea-4b3d-908f-1e7cb0825861   76Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-4    Bound    pvc-d9ff4886-2714-442d-b7f0-e309d28a8762   144Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-5    Bound    pvc-adbf4f19-a139-442c-8530-b84cc8af44bf   107Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-6    Bound    pvc-1426a615-1205-42e2-a8a2-fb5a82a77e3f   123Gi      RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-7    Bound    pvc-0d97073d-75fb-4d88-8782-7c2b4114fa66   90Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-8    Bound    pvc-893f0035-2d17-4b3c-bdc6-01347106d592   91Gi       RWO            ocs-storagecluster-ceph-rbd   39h
persistentvolumeclaim/busybox-pvc-9    Bound    pvc-a47b6da0-cb62-40da-9a3a-6d7cb055a4bc   111Gi      RWO            ocs-storagecluster-ceph-rbd   39h

must gather
c1 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-2123501/must-gather.local.4695170007967000175-c1/

c2 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-2123501/must-gather.local.4636546916742256672-c2/

hub- http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/BZ-2123501/must-gather.local.3962593501856934386-hub/


verified on - 4.12.0-167
ACM - 2.7
Additional Info - 
     - Did worker and master node down tests
Comment 46 errata-xmlrpc 2023-03-08 15:31:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.12.1 security bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:1170