Description of problem (please be detailed as possible and provide log snippests): Version of all relevant components (if applicable): ODF 4.14.0-132.stable OCP 4.14.0-0.nightly-2023-09-02-132842 ACM 2.9.0-DOWNSTREAM-2023-08-24-09-30-12 ceph version 17.2.6-138.el9cp (b488c8dad42b2ecffcd96f3d76eeeecce48b8590) quincy (stable) Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. On a RDR setup, deploy app-set and subscription based workloads on both the managed clusters and run IOs for a few days. 2. Ensure submariner (subctl verify) check passes and mirroring is properly established. 3. Then delete all the workloads one by one from ACM UI and ensure deletion is successful. Actual results: App-set workload resources remain stuck on deletion From Hub- Before deletion: amagrawa:~$ oc get placement -A NAMESPACE NAME SUCCEEDED REASON SELECTEDCLUSTERS busybox-workloads-2 busybox-workloads-2-placement-1 True AllDecisionsScheduled 1 busybox-workloads-5 busybox-workloads-5-placement-1 True AllDecisionsScheduled 1 busybox-workloads-6 busybox-workloads-6-placement-1 True AllDecisionsScheduled 1 busybox-workloads-6 busybox-workloads-6-placement-2 True AllDecisionsScheduled 1 openshift-gitops all-openshift-clusters True AllDecisionsScheduled 2 openshift-gitops busybox-workloads-1-placement True AllDecisionsScheduled 1 openshift-gitops busybox-workloads-3-placement True AllDecisionsScheduled 1 openshift-gitops busybox-workloads-4-placement True AllDecisionsScheduled 1 amagrawa:~$ oc get applications -A NAMESPACE NAME TYPE VERSION OWNER READY AGE busybox-workloads-2 busybox-workloads-2 3d18h busybox-workloads-5 busybox-workloads-5 26h busybox-workloads-6 busybox-workloads-6 4h37m amagrawa:~$ oc get placementrule -A NAMESPACE NAME AGE REPLICAS busybox-workloads-5 vs-secret-c8373780841d23d4acbb2a67f263fea8 26h openshift-gitops busybox-workloads-3-placement-drpc-vs-secret 3d18h openshift-operators plrule-0b269ab0ff4d7575a448121fd02f8a61b7f3871 3d21h openshift-operators plrule-6e560664086c1780c2df286715aa49143c435f3 3d21h After deletion: amagrawa:~$ drpc NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-5 busybox-workloads-5-placement-1-drpc 26h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-3-placement-drpc 3d18h amagrawa-c1 Deployed Completed True amagrawa:~$ oc get placementrule -A NAMESPACE NAME AGE REPLICAS busybox-workloads-5 vs-secret-c8373780841d23d4acbb2a67f263fea8 26h openshift-gitops busybox-workloads-3-placement-drpc-vs-secret 3d18h openshift-operators plrule-0b269ab0ff4d7575a448121fd02f8a61b7f3871 3d21h openshift-operators plrule-6e560664086c1780c2df286715aa49143c435f3 3d21h amagrawa:~$ oc get applications -A NAMESPACE NAME TYPE VERSION OWNER READY AGE busybox-workloads-5 busybox-workloads-5 26h amagrawa:~$ oc get placement -A NAMESPACE NAME SUCCEEDED REASON SELECTEDCLUSTERS busybox-workloads-5 busybox-workloads-5-placement-1 True AllDecisionsScheduled 1 openshift-gitops all-openshift-clusters True AllDecisionsScheduled 2 openshift-gitops busybox-workloads-3-placement True AllDecisionsScheduled 1 2 app-set based workload has PVCs,VRs and VRGs remaining as deletiontimestamp wasn't set on VRG but the drpc, placement and application was deleted for those 2 apps. From C1- amagrawa:~$ busybox-1 Now using project "busybox-workloads-1" on server "https://api.amagrawa-c1.qe.rh-ocs.com:6443". NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE persistentvolumeclaim/dd-io-pvc-1 Terminating pvc-91d88997-c738-474c-9058-41c6f80f96d4 117Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-2 Terminating pvc-15000553-3f26-4619-9108-55c89b0e567a 143Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-3 Terminating pvc-443d7850-ee01-4325-a769-adb1471cfd1c 134Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-4 Terminating pvc-f6ccee07-c945-4ac7-a583-0cfba636508c 106Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-5 Terminating pvc-547db850-36fd-4c63-a248-3f64c76fabfb 115Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-6 Terminating pvc-38cff2c5-13f6-47de-8bd5-dfb57cb1595c 129Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem persistentvolumeclaim/dd-io-pvc-7 Terminating pvc-540c2bde-acf0-4297-a4b6-eae06a5f6816 149Gi RWO ocs-storagecluster-ceph-rbd 3d22h Filesystem NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE volumereplication.replication.storage.openshift.io/dd-io-pvc-1 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-1 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-2 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-2 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-3 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-3 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-4 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-4 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-5 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-5 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-6 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-6 primary Primary volumereplication.replication.storage.openshift.io/dd-io-pvc-7 3d22h rbd-volumereplicationclass-539797778 dd-io-pvc-7 primary Primary NAME DESIREDSTATE CURRENTSTATE volumereplicationgroup.ramendr.openshift.io/busybox-workloads-1-placement-drpc primary Primary From C2- amagrawa:c2$ busybox-4 Already on project "busybox-workloads-4" on server "https://api.amagrawa-c2.qe.rh-ocs.com:6443". NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE persistentvolumeclaim/busybox-pvc-41 Terminating pvc-7134ea6b-3b06-44d9-8fba-4cf1456caaf3 42Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-42 Terminating pvc-ac4c888f-57fb-4587-9b24-d8d170b91107 81Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-43 Terminating pvc-cf760066-7a5b-4ac2-a674-22fd36f7e078 28Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-44 Terminating pvc-c0373692-7955-4b1b-a75e-f7df477e8486 118Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-45 Terminating pvc-4aa5855b-f650-4a4d-93ff-9e012d02c487 19Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-46 Terminating pvc-8a40bf12-e6a4-4584-ab0b-4f599bc9f2b3 129Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-47 Terminating pvc-2fbbe1c6-6e29-4946-954d-673e020b4094 43Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-48 Terminating pvc-5d438cdb-3348-4c16-808a-263d631a3b33 57Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-49 Terminating pvc-49497908-217f-4149-85b4-c1b7cb1e5537 89Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-50 Terminating pvc-ca74a821-934c-4d80-93d2-55ff4870b292 124Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-51 Terminating pvc-b4bb26bc-417b-434e-ad0e-78ccb50860a6 95Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-52 Terminating pvc-bdf6b3d8-6232-45de-8b2f-0136049b9d1a 129Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-53 Terminating pvc-5faa1227-1dcf-4266-a587-4b310d7b5531 51Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-54 Terminating pvc-7a65522e-6bb4-4094-b386-60523588d9a5 30Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-55 Terminating pvc-10c5b607-874c-4280-972c-9dc9eb201508 102Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-56 Terminating pvc-b804b4e1-381c-499d-82e8-f1bf18da160d 40Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-57 Terminating pvc-a6a8a86c-2bf9-4041-a6cd-57d03654a6fa 146Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-58 Terminating pvc-ed5e9d0b-8bc9-46c1-8b0c-e28cd982b5e5 63Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-59 Terminating pvc-b90a5cb6-a6fd-4487-bdae-baec56b80b20 118Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem persistentvolumeclaim/busybox-pvc-60 Terminating pvc-b5d5eeff-9fb3-4634-95a3-db89aeb95554 25Gi RWO ocs-storagecluster-ceph-rbd 44h Filesystem NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE volumereplication.replication.storage.openshift.io/busybox-pvc-41 44h rbd-volumereplicationclass-1626052323 busybox-pvc-41 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-42 44h rbd-volumereplicationclass-1626052323 busybox-pvc-42 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-43 44h rbd-volumereplicationclass-1626052323 busybox-pvc-43 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-44 44h rbd-volumereplicationclass-1626052323 busybox-pvc-44 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-45 44h rbd-volumereplicationclass-1626052323 busybox-pvc-45 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-46 44h rbd-volumereplicationclass-1626052323 busybox-pvc-46 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-47 44h rbd-volumereplicationclass-1626052323 busybox-pvc-47 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-48 44h rbd-volumereplicationclass-1626052323 busybox-pvc-48 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-49 44h rbd-volumereplicationclass-1626052323 busybox-pvc-49 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-50 44h rbd-volumereplicationclass-1626052323 busybox-pvc-50 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-51 44h rbd-volumereplicationclass-1626052323 busybox-pvc-51 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-52 44h rbd-volumereplicationclass-1626052323 busybox-pvc-52 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-53 44h rbd-volumereplicationclass-1626052323 busybox-pvc-53 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-54 44h rbd-volumereplicationclass-1626052323 busybox-pvc-54 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-55 44h rbd-volumereplicationclass-1626052323 busybox-pvc-55 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-56 44h rbd-volumereplicationclass-1626052323 busybox-pvc-56 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-57 44h rbd-volumereplicationclass-1626052323 busybox-pvc-57 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-58 44h rbd-volumereplicationclass-1626052323 busybox-pvc-58 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-59 44h rbd-volumereplicationclass-1626052323 busybox-pvc-59 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-60 44h rbd-volumereplicationclass-1626052323 busybox-pvc-60 primary Primary NAME DESIREDSTATE CURRENTSTATE volumereplicationgroup.ramendr.openshift.io/busybox-workloads-4-placement-drpc primary Primary Cleanup of all subscription based workloads was successful. Expected results: PVCs, VRs and VRs for app-set based workloads should be cleaned. It is expected that deletion timestamp is set on VRG as placement and drpc for busybox-workloads-1 and busybox-workloads-4 were deleted. Additional info:
If we delete the manifestwork from hub, the stuck resources gets cleaned up (VRGs, VRs and PVCs along with rbd mirror images) from the managed clusters. amagrawa:~$ oc delete manifestwork -n amagrawa-c1 busybox-workloads-1-placement-drpc-busybox-workloads-1-vrg-mw manifestwork.work.open-cluster-management.io "busybox-workloads-1-placement-drpc-busybox-workloads-1-vrg-mw" deleted amagrawa:~$ oc delete manifestwork -n amagrawa-c2 busybox-workloads-4-placement-drpc-busybox-workloads-4-vrg-mw manifestwork.work.open-cluster-management.io "busybox-workloads-4-placement-drpc-busybox-workloads-4-vrg-mw" deleted The suggested workaround seems to be working perfectly fine.
This issue is hit every time appset based workloads are deleted and probably is also a regression.
Marking it as a regression as this behaviour has changed b/w 4.13 and 4.14.
VERIFICATION COMMENTS ===================== Steps to Reproduce: 1. On a RDR setup, deploy app-set and subscription based workloads on both the managed clusters and run IOs for a few days. 2. Then delete all the workloads one by one from ACM UI and ensure deletion is successful. Actual results: App-set workload resources remain stuck on deletion Expected results: PVCs, VRs and VRs for app-set based workloads should be cleaned. ________________________________________________________________ Output before the deletion of the application ============================================= on C1 ----- oc get pods NAME READY STATUS RESTARTS AGE busybox-1-7f7bf8c5d9-9x9w4 1/1 Running 0 64m busybox-10-7b7bddddf8-p7p4g 1/1 Running 0 64m busybox-11-6c4cf4bfb-25nss 1/1 Running 0 64m busybox-12-7968f7d4bb-67825 1/1 Running 0 64m busybox-13-674b97b564-kmpt5 1/1 Running 0 64m busybox-14-f59899658-lznkf 1/1 Running 0 64m busybox-15-867dd79cbd-44d5r 1/1 Running 0 64m busybox-16-866d576d54-mp4vm 1/1 Running 0 64m busybox-17-8d7df8b76-wlsnz 1/1 Running 0 64m busybox-18-75cdf6f4c4-fvrdm 1/1 Running 0 64m busybox-19-6bcbc84d68-mmz68 1/1 Running 0 64m busybox-2-5cffb67686-znd2s 1/1 Running 0 64m busybox-20-fdbd78dbd-7sm85 1/1 Running 0 64m busybox-3-7ffc7c8fbb-9ggxm 1/1 Running 0 64m busybox-4-66688c494b-877fc 1/1 Running 0 64m busybox-5-56978ff94-2d7rn 1/1 Running 0 64m busybox-6-57544b458b-4gtdx 1/1 Running 0 64m busybox-7-77ff998b8b-bjz5f 1/1 Running 0 64m busybox-8-6d5cdc5678-x4vsz 1/1 Running 0 64m busybox-9-79c789995d-9sg4z 1/1 Running 0 64m oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE busybox-pvc-1 Bound pvc-017d18f9-1ea8-4698-878e-98680af606ca 94Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-10 Bound pvc-2613d35d-7f26-4fcf-9c9f-50ba1537a9c5 87Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-11 Bound pvc-6bdd9e1c-f190-40c3-b6f9-f60c6fda752f 33Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-12 Bound pvc-57c98c96-6d79-4fd4-a4e5-e4fdc0e06739 147Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-13 Bound pvc-f83c2819-2b5c-4bc3-b8c4-bcecef2c06c1 77Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-14 Bound pvc-f0b1f779-6994-4321-90ea-880acfa49ca8 70Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-15 Bound pvc-4e6e3db2-de2e-45ef-97f2-4354b62ef329 131Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-16 Bound pvc-90543b2f-e392-4876-9d83-08dc72507fcb 127Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-17 Bound pvc-6aa594e2-2559-4fa6-8049-bd5d4c4dd4df 58Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-18 Bound pvc-9cb5eca4-0d5f-4a0c-99f7-b4b58d9111c5 123Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-19 Bound pvc-cabad17d-922a-4daa-a0ba-ac3755790252 61Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-2 Bound pvc-7b2e23c7-cea6-4c44-a3da-980eadd0877a 44Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-20 Bound pvc-5b0adeaa-2223-4ae3-9603-8e6796c38917 33Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-3 Bound pvc-766ef282-55da-472c-9053-46d6fceef058 76Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-4 Bound pvc-1ea5c670-8eeb-4600-b77f-217e04e1d48f 144Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-5 Bound pvc-bf525325-9f25-4079-83ee-29960df4f589 107Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-6 Bound pvc-4c5a9f3d-0e35-43e4-a01a-138402ebc6b3 123Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-7 Bound pvc-371c5d4b-c68e-4b43-9be1-ce4b87ab13ed 90Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-8 Bound pvc-572f2e83-75b0-469d-a53c-64701ae4e191 91Gi RWO ocs-storagecluster-ceph-rbd 64m busybox-pvc-9 Bound pvc-e12a4a31-a40d-4d7b-bea9-c5cf4b0b10b9 111Gi RWO ocs-storagecluster-ceph-rbd 64m $ oc get vr,vrg NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE volumereplication.replication.storage.openshift.io/busybox-pvc-1 5m58s rbd-volumereplicationclass-1625360775 busybox-pvc-1 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-10 6m16s rbd-volumereplicationclass-1625360775 busybox-pvc-10 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-11 6m14s rbd-volumereplicationclass-1625360775 busybox-pvc-11 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-12 5m55s rbd-volumereplicationclass-1625360775 busybox-pvc-12 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-13 5m52s rbd-volumereplicationclass-1625360775 busybox-pvc-13 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-14 6m13s rbd-volumereplicationclass-1625360775 busybox-pvc-14 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-15 5m50s rbd-volumereplicationclass-1625360775 busybox-pvc-15 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-16 5m45s rbd-volumereplicationclass-1625360775 busybox-pvc-16 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-17 6m12s rbd-volumereplicationclass-1625360775 busybox-pvc-17 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-18 6m10s rbd-volumereplicationclass-1625360775 busybox-pvc-18 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-19 5m39s rbd-volumereplicationclass-1625360775 busybox-pvc-19 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-2 5m38s rbd-volumereplicationclass-1625360775 busybox-pvc-2 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-20 6m9s rbd-volumereplicationclass-1625360775 busybox-pvc-20 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-3 6m7s rbd-volumereplicationclass-1625360775 busybox-pvc-3 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-4 6m rbd-volumereplicationclass-1625360775 busybox-pvc-4 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-5 6m4s rbd-volumereplicationclass-1625360775 busybox-pvc-5 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-6 5m36s rbd-volumereplicationclass-1625360775 busybox-pvc-6 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-7 6m13s rbd-volumereplicationclass-1625360775 busybox-pvc-7 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-8 5m34s rbd-volumereplicationclass-1625360775 busybox-pvc-8 primary Primary volumereplication.replication.storage.openshift.io/busybox-pvc-9 6m1s rbd-volumereplicationclass-1625360775 busybox-pvc-9 primary Primary NAME DESIREDSTATE CURRENTSTATE volumereplicationgroup.ramendr.openshift.io/busybox-1-placement-drpc primary Primary sh-5.1$ rbd mirror snapshot schedule list --recursive POOL NAMESPACE IMAGE SCHEDULE ocs-storagecluster-cephblockpool csi-vol-08fb33b0-a0ac-46b0-8e90-2f0cbdfe96e9 every 5m ocs-storagecluster-cephblockpool csi-vol-10c9c6a3-e1a0-41f6-ad2f-fc1e14c46a41 every 5m ocs-storagecluster-cephblockpool csi-vol-251d1c3e-98a0-4bb1-a3b0-79b7e0c31376 every 5m ocs-storagecluster-cephblockpool csi-vol-2806f79c-4e8f-48cf-a556-facd87cecd6d every 5m ocs-storagecluster-cephblockpool csi-vol-2a1366df-30ac-4b7c-8369-e8523ea09e5b every 5m ocs-storagecluster-cephblockpool csi-vol-2c60e8e7-37f2-474f-a91a-1c64f96816df every 5m ocs-storagecluster-cephblockpool csi-vol-30745faa-b699-45ec-b264-1f838e23b508 every 5m ocs-storagecluster-cephblockpool csi-vol-333a7648-9a3e-4f6c-a523-694c6f047211 every 5m ocs-storagecluster-cephblockpool csi-vol-4d8da735-173a-4b8a-af9c-64f24c731bae every 5m ocs-storagecluster-cephblockpool csi-vol-6a59c653-55fc-4c58-80d9-bb509a0ff3ce every 5m ocs-storagecluster-cephblockpool csi-vol-790ef47e-9251-4481-ad30-67a2eb0bcb30 every 5m ocs-storagecluster-cephblockpool csi-vol-8a6a021f-4591-42b6-91ca-c788abc6daff every 5m ocs-storagecluster-cephblockpool csi-vol-ab7abe94-a8d5-4224-af11-401c2667aa63 every 5m ocs-storagecluster-cephblockpool csi-vol-d0889f52-907e-4843-bb19-b61afd0755e8 every 5m ocs-storagecluster-cephblockpool csi-vol-d3b762cc-c0b9-4426-8911-40e4df5698d1 every 5m ocs-storagecluster-cephblockpool csi-vol-d45dd1f8-59ce-4b7a-9965-d8c08007a49b every 5m ocs-storagecluster-cephblockpool csi-vol-d5d42fe3-d883-4d42-801c-164e4096f3a6 every 5m ocs-storagecluster-cephblockpool csi-vol-e0d50304-b811-4c55-b6d4-570dbb874dc4 every 5m ocs-storagecluster-cephblockpool csi-vol-ee765cc3-119c-4c49-b457-d9335e235a42 every 5m ocs-storagecluster-cephblockpool csi-vol-f58f166f-53e4-448e-8cb2-985502b2bb89 every 5m sh-5.1$ for i in $(rbd ls -p ocs-storagecluster-cephblockpool); do echo $i; rbd snap ls ocs-storagecluster-cephblockpool/$i --all 2>/dev/null; echo "##########################################";done csi-vol-08fb33b0-a0ac-46b0-8e90-2f0cbdfe96e9 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 20 .mirror.primary.8681957e-fdbf-46ba-b998-3f06d9147396.7f9040ec-5c78-4a2a-81d8-87819e0a9415 61 GiB Fri Oct 20 09:09:53 2023 mirror (primary peer_uuids:[]) 55 .mirror.primary.8681957e-fdbf-46ba-b998-3f06d9147396.0c2e6d1f-99d4-4e7b-aa13-90000e248527 61 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-0a9ddb11-5c8c-411e-abb1-b4aa4ca2be68 ########################################## csi-vol-10c9c6a3-e1a0-41f6-ad2f-fc1e14c46a41 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 24 .mirror.primary.51b79c72-d3ff-4325-b041-c7262cd2adf6.e33ff94a-4f78-4c02-9fc4-456c73b8afe5 70 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 37 .mirror.primary.51b79c72-d3ff-4325-b041-c7262cd2adf6.ab4fbdfa-9a76-491a-acbf-d3f91b69c25b 70 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-251d1c3e-98a0-4bb1-a3b0-79b7e0c31376 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 26 .mirror.primary.0b1b9200-0b49-4f19-9417-9695eeb52ff4.11b2954f-3369-4ef1-b43e-f8c452f8791c 87 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 35 .mirror.primary.0b1b9200-0b49-4f19-9417-9695eeb52ff4.ee821fdd-f9f1-462e-9593-8303bca74325 87 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-2806f79c-4e8f-48cf-a556-facd87cecd6d SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 22 .mirror.primary.a1342bd0-98ab-4511-bf5a-98257fa1b8df.1d73eb87-b023-4c60-a8d8-a728526f0871 123 GiB Fri Oct 20 09:09:56 2023 mirror (primary peer_uuids:[]) 39 .mirror.primary.a1342bd0-98ab-4511-bf5a-98257fa1b8df.118fb5ec-6f27-4a7b-8334-af1d2c51b222 123 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-2a1366df-30ac-4b7c-8369-e8523ea09e5b SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 13 .mirror.primary.9e1d65d9-d20b-48db-8e46-b0538ab18531.ea21d643-48ab-4361-bd39-61b670238ed4 111 GiB Fri Oct 20 09:09:31 2023 mirror (primary peer_uuids:[]) 52 .mirror.primary.9e1d65d9-d20b-48db-8e46-b0538ab18531.4b7e2aef-5562-4c76-80b0-04eb320fe307 111 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-2c60e8e7-37f2-474f-a91a-1c64f96816df SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 14 .mirror.primary.c778e669-7451-4bc0-bab3-3cba2ed6aa30.24fbae01-492d-403e-94f6-ae92736e096d 144 GiB Fri Oct 20 09:09:33 2023 mirror (primary peer_uuids:[]) 51 .mirror.primary.c778e669-7451-4bc0-bab3-3cba2ed6aa30.e6109ce3-1be8-4c75-be3b-308ac049c478 144 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-30745faa-b699-45ec-b264-1f838e23b508 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 28 .mirror.primary.a8a12e9e-08fe-4b67-b9b1-909c1c0b0d61.00295a14-3893-427e-9e3c-3dc5c5240cd0 90 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 33 .mirror.primary.a8a12e9e-08fe-4b67-b9b1-909c1c0b0d61.2ca912be-2c0e-4caa-b325-c1ac3aef1168 90 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-333a7648-9a3e-4f6c-a523-694c6f047211 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 15 .mirror.primary.a5d9ce77-1b59-4cdb-a154-214adb791ac5.b74d3565-a1e9-4e80-ae8b-db65a4787f29 94 GiB Fri Oct 20 09:09:35 2023 mirror (primary peer_uuids:[]) 49 .mirror.primary.a5d9ce77-1b59-4cdb-a154-214adb791ac5.4b438988-3043-4865-bcfd-5b9c49ed33f6 94 GiB Fri Oct 20 09:15:02 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-4d8da735-173a-4b8a-af9c-64f24c731bae SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 19 .mirror.primary.e2811ccb-7f19-4885-8051-b1e407e9dfe3.60da9f13-6707-479f-91df-d38ed2dcc851 127 GiB Fri Oct 20 09:09:50 2023 mirror (primary peer_uuids:[]) 53 .mirror.primary.e2811ccb-7f19-4885-8051-b1e407e9dfe3.a1e941d5-9d7d-48fa-ac75-0a7c10b089b5 127 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-6a59c653-55fc-4c58-80d9-bb509a0ff3ce SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 18 .mirror.primary.8abf41cb-b5ba-4bc3-bdbe-9f56fe730fef.902836e2-a573-45ee-9c6d-2146a2e6e6df 131 GiB Fri Oct 20 09:09:45 2023 mirror (primary peer_uuids:[]) 38 .mirror.primary.8abf41cb-b5ba-4bc3-bdbe-9f56fe730fef.6846d9cc-c4ea-4740-9da7-f03df3ea8e70 131 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-790ef47e-9251-4481-ad30-67a2eb0bcb30 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 27 .mirror.primary.0e1b8900-9e6e-493d-a9da-88a168026b6a.25033485-07da-4b7e-8f25-957266b74ee2 33 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 34 .mirror.primary.0e1b8900-9e6e-493d-a9da-88a168026b6a.edad0992-bed0-496c-9f0a-83abb044ffcd 33 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-8a6a021f-4591-42b6-91ca-c788abc6daff SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 30 .mirror.primary.4810a66b-ba0c-4e36-a12f-2b058e541980.2351407f-ebe5-4204-8ca0-2fd5a56b9a0f 33 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 32 .mirror.primary.4810a66b-ba0c-4e36-a12f-2b058e541980.47468c20-06aa-4e9b-9941-0730a107da8e 33 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-ab7abe94-a8d5-4224-af11-401c2667aa63 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 23 .mirror.primary.11ad065f-d7f0-4eed-8720-c91871b1f906.1ace666a-1e6f-493e-8d9c-132197f17862 91 GiB Fri Oct 20 09:09:57 2023 mirror (primary peer_uuids:[]) 56 .mirror.primary.11ad065f-d7f0-4eed-8720-c91871b1f906.73800b65-835e-44fc-a2eb-22129de47733 91 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-d0889f52-907e-4843-bb19-b61afd0755e8 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 25 .mirror.primary.d230347b-cfe5-4c2f-badf-99116e2945be.5a5e95df-cbbe-411f-bc7a-042e7bf1ce4b 123 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 31 .mirror.primary.d230347b-cfe5-4c2f-badf-99116e2945be.5095cd14-abce-4d8e-ba39-447b0e97ec50 123 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-d3b762cc-c0b9-4426-8911-40e4df5698d1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 11 .mirror.primary.3b5d1032-0cb2-4de3-b6ff-b80ab724e424.8d31568d-25b6-4124-b7cb-9aae16db0ba1 76 GiB Fri Oct 20 09:09:26 2023 mirror (primary peer_uuids:[]) 54 .mirror.primary.3b5d1032-0cb2-4de3-b6ff-b80ab724e424.a980aa45-2300-4143-b2ae-c87c7abcff73 76 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-d45dd1f8-59ce-4b7a-9965-d8c08007a49b SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 12 .mirror.primary.50f53763-cf79-4300-8593-22bfc4a94f05.8651e61a-0638-426f-92ae-d451f7a33d5a 107 GiB Fri Oct 20 09:09:29 2023 mirror (primary peer_uuids:[]) 40 .mirror.primary.50f53763-cf79-4300-8593-22bfc4a94f05.1667c213-0102-4f5b-b48c-996bedb63e7b 107 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-d5d42fe3-d883-4d42-801c-164e4096f3a6 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 16 .mirror.primary.8b3ca4cd-e375-4c1e-ad73-163e99b4c960.b9a89fcb-bf58-4b9f-a971-557356737eaf 147 GiB Fri Oct 20 09:09:38 2023 mirror (primary peer_uuids:[]) 47 .mirror.primary.8b3ca4cd-e375-4c1e-ad73-163e99b4c960.c3af8418-1631-424e-af02-fca486801d75 147 GiB Fri Oct 20 09:15:02 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-e0d50304-b811-4c55-b6d4-570dbb874dc4 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 17 .mirror.primary.6f24657c-4ba8-43d9-99d6-3368632c6598.c7f97870-3f8c-4638-a26c-22c0f87b655b 77 GiB Fri Oct 20 09:09:41 2023 mirror (primary peer_uuids:[]) 48 .mirror.primary.6f24657c-4ba8-43d9-99d6-3368632c6598.b7381c44-6ac0-4ec3-8833-fa9ec3c7f0b2 77 GiB Fri Oct 20 09:15:02 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-ee765cc3-119c-4c49-b457-d9335e235a42 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 21 .mirror.primary.7c0489e9-9767-4e64-b0f2-b04e041d36dc.e48fdbd8-207f-4a5f-92aa-02ea83c90329 44 GiB Fri Oct 20 09:09:54 2023 mirror (primary peer_uuids:[]) 57 .mirror.primary.7c0489e9-9767-4e64-b0f2-b04e041d36dc.5500bb95-0ec2-491e-a658-4a69b58cf394 44 GiB Fri Oct 20 09:15:03 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## csi-vol-f58f166f-53e4-448e-8cb2-985502b2bb89 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 29 .mirror.primary.7f007c71-a25a-45ed-96f6-a99ee2104593.b3ddd232-ba06-4608-b0de-c6989d750ac8 58 GiB Fri Oct 20 09:10:03 2023 mirror (primary peer_uuids:[]) 36 .mirror.primary.7f007c71-a25a-45ed-96f6-a99ee2104593.e79603db-85fc-4822-b171-3be442d1989e 58 GiB Fri Oct 20 09:15:00 2023 mirror (primary peer_uuids:[0dfdf75a-0ec8-4db7-a58e-3ff2449ba9f5]) ########################################## $ rbd mirror snapshot schedule status SCHEDULE TIME IMAGE 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-d0889f52-907e-4843-bb19-b61afd0755e8 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-8a6a021f-4591-42b6-91ca-c788abc6daff 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-251d1c3e-98a0-4bb1-a3b0-79b7e0c31376 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-f58f166f-53e4-448e-8cb2-985502b2bb89 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-790ef47e-9251-4481-ad30-67a2eb0bcb30 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-10c9c6a3-e1a0-41f6-ad2f-fc1e14c46a41 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-30745faa-b699-45ec-b264-1f838e23b508 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-6a59c653-55fc-4c58-80d9-bb509a0ff3ce 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-2806f79c-4e8f-48cf-a556-facd87cecd6d 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-d45dd1f8-59ce-4b7a-9965-d8c08007a49b 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-e0d50304-b811-4c55-b6d4-570dbb874dc4 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-d5d42fe3-d883-4d42-801c-164e4096f3a6 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-333a7648-9a3e-4f6c-a523-694c6f047211 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-2a1366df-30ac-4b7c-8369-e8523ea09e5b 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-2c60e8e7-37f2-474f-a91a-1c64f96816df 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-4d8da735-173a-4b8a-af9c-64f24c731bae 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-08fb33b0-a0ac-46b0-8e90-2f0cbdfe96e9 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-d3b762cc-c0b9-4426-8911-40e4df5698d1 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-ab7abe94-a8d5-4224-af11-401c2667aa63 2023-10-20 09:45:00 ocs-storagecluster-cephblockpool/csi-vol-ee765cc3-119c-4c49-b457-d9335e235a42 $ rbd mirror snapshot schedule list --recursive POOL NAMESPACE IMAGE SCHEDULE ocs-storagecluster-cephblockpool csi-vol-08fb33b0-a0ac-46b0-8e90-2f0cbdfe96e9 every 5m ocs-storagecluster-cephblockpool csi-vol-10c9c6a3-e1a0-41f6-ad2f-fc1e14c46a41 every 5m ocs-storagecluster-cephblockpool csi-vol-251d1c3e-98a0-4bb1-a3b0-79b7e0c31376 every 5m ocs-storagecluster-cephblockpool csi-vol-2806f79c-4e8f-48cf-a556-facd87cecd6d every 5m ocs-storagecluster-cephblockpool csi-vol-2a1366df-30ac-4b7c-8369-e8523ea09e5b every 5m ocs-storagecluster-cephblockpool csi-vol-2c60e8e7-37f2-474f-a91a-1c64f96816df every 5m ocs-storagecluster-cephblockpool csi-vol-30745faa-b699-45ec-b264-1f838e23b508 every 5m ocs-storagecluster-cephblockpool csi-vol-333a7648-9a3e-4f6c-a523-694c6f047211 every 5m ocs-storagecluster-cephblockpool csi-vol-4d8da735-173a-4b8a-af9c-64f24c731bae every 5m ocs-storagecluster-cephblockpool csi-vol-6a59c653-55fc-4c58-80d9-bb509a0ff3ce every 5m ocs-storagecluster-cephblockpool csi-vol-790ef47e-9251-4481-ad30-67a2eb0bcb30 every 5m ocs-storagecluster-cephblockpool csi-vol-8a6a021f-4591-42b6-91ca-c788abc6daff every 5m ocs-storagecluster-cephblockpool csi-vol-ab7abe94-a8d5-4224-af11-401c2667aa63 every 5m ocs-storagecluster-cephblockpool csi-vol-d0889f52-907e-4843-bb19-b61afd0755e8 every 5m ocs-storagecluster-cephblockpool csi-vol-d3b762cc-c0b9-4426-8911-40e4df5698d1 every 5m ocs-storagecluster-cephblockpool csi-vol-d45dd1f8-59ce-4b7a-9965-d8c08007a49b every 5m ocs-storagecluster-cephblockpool csi-vol-d5d42fe3-d883-4d42-801c-164e4096f3a6 every 5m ocs-storagecluster-cephblockpool csi-vol-e0d50304-b811-4c55-b6d4-570dbb874dc4 every 5m ocs-storagecluster-cephblockpool csi-vol-ee765cc3-119c-4c49-b457-d9335e235a42 every 5m ocs-storagecluster-cephblockpool csi-vol-f58f166f-53e4-448e-8cb2-985502b2bb89 every 5m $ rbd mirror pool status ocs-storagecluster-cephblockpool health: OK daemon health: OK image health: OK images: 20 total 20 replaying On C2 ------ $ rbd mirror pool status ocs-storagecluster-cephblockpool health: OK daemon health: OK image health: OK images: 20 total 20 replaying for i in $(rbd ls -p ocs-storagecluster-cephblockpool); do echo $i; rbd snap ls ocs-storagecluster-cephblockpool/$i --all 2>/dev/null; echo "##########################################";done csi-vol-08fb33b0-a0ac-46b0-8e90-2f0cbdfe96e9 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 137 .mirror.non_primary.8681957e-fdbf-46ba-b998-3f06d9147396.024330b7-0516-41d1-9b11-8c126be19a24 61 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:125 copied) ########################################## csi-vol-10c9c6a3-e1a0-41f6-ad2f-fc1e14c46a41 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 118 .mirror.non_primary.51b79c72-d3ff-4325-b041-c7262cd2adf6.7aaab078-157b-4e64-b4b7-00eda001a611 70 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:102 copied) ########################################## csi-vol-251d1c3e-98a0-4bb1-a3b0-79b7e0c31376 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 126 .mirror.non_primary.0b1b9200-0b49-4f19-9417-9695eeb52ff4.33dee69b-8309-4bd2-a1e6-75ec757aea91 87 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:101 copied) ########################################## csi-vol-2806f79c-4e8f-48cf-a556-facd87cecd6d SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 120 .mirror.non_primary.a1342bd0-98ab-4511-bf5a-98257fa1b8df.855b00d2-4cf8-463c-8cb5-39aead5451e0 123 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:105 copied) ########################################## csi-vol-2a1366df-30ac-4b7c-8369-e8523ea09e5b SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 128 .mirror.non_primary.9e1d65d9-d20b-48db-8e46-b0538ab18531.4bb1e3b4-0672-4946-a58b-a5d59a68b856 111 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:119 copied) ########################################## csi-vol-2c60e8e7-37f2-474f-a91a-1c64f96816df SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 133 .mirror.non_primary.c778e669-7451-4bc0-bab3-3cba2ed6aa30.8ee91931-3ef9-472a-8fc2-06d0a3c0eef9 144 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:126 copied) ########################################## csi-vol-30745faa-b699-45ec-b264-1f838e23b508 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 127 .mirror.non_primary.a8a12e9e-08fe-4b67-b9b1-909c1c0b0d61.079826e4-442b-4c57-827f-4f6bbfee5ea8 90 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:104 copied) ########################################## csi-vol-333a7648-9a3e-4f6c-a523-694c6f047211 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 136 .mirror.non_primary.a5d9ce77-1b59-4cdb-a154-214adb791ac5.915cb6d0-4610-4cdf-b5bf-1e29c2b6e788 94 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:120 copied) ########################################## csi-vol-4d8da735-173a-4b8a-af9c-64f24c731bae SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 134 .mirror.non_primary.e2811ccb-7f19-4885-8051-b1e407e9dfe3.ec51a98b-5fe1-4d98-98a4-91f634f227ea 127 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:121 copied) ########################################## csi-vol-6a59c653-55fc-4c58-80d9-bb509a0ff3ce SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 125 .mirror.non_primary.8abf41cb-b5ba-4bc3-bdbe-9f56fe730fef.e2d417ca-a1ae-4724-9072-3919dd856083 131 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:106 copied) ########################################## csi-vol-78a5f290-5d7f-4f95-ac62-7370f64b9770 ########################################## csi-vol-790ef47e-9251-4481-ad30-67a2eb0bcb30 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 123 .mirror.non_primary.0e1b8900-9e6e-493d-a9da-88a168026b6a.04bb4664-0966-47c8-816a-f9f68073773d 33 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:103 copied) ########################################## csi-vol-8a6a021f-4591-42b6-91ca-c788abc6daff SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 124 .mirror.non_primary.4810a66b-ba0c-4e36-a12f-2b058e541980.66fa51b8-e1a0-4872-aa15-ee9180d63233 33 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:100 copied) ########################################## csi-vol-ab7abe94-a8d5-4224-af11-401c2667aa63 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 132 .mirror.non_primary.11ad065f-d7f0-4eed-8720-c91871b1f906.1c37e3ec-e950-42cf-ae04-fe6c3fab4692 91 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:127 copied) ########################################## csi-vol-d0889f52-907e-4843-bb19-b61afd0755e8 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 122 .mirror.non_primary.d230347b-cfe5-4c2f-badf-99116e2945be.8e4fe6bf-2c38-4907-bf52-e65a673df299 123 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:98 copied) ########################################## csi-vol-d3b762cc-c0b9-4426-8911-40e4df5698d1 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 131 .mirror.non_primary.3b5d1032-0cb2-4de3-b6ff-b80ab724e424.3d422ff2-4b13-4844-bc99-bd3b71bb49e6 76 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:124 copied) ########################################## csi-vol-d45dd1f8-59ce-4b7a-9965-d8c08007a49b SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 121 .mirror.non_primary.50f53763-cf79-4300-8593-22bfc4a94f05.a50585ee-876f-4bcf-9a37-68dd3072499e 107 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:107 copied) ########################################## csi-vol-d5d42fe3-d883-4d42-801c-164e4096f3a6 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 135 .mirror.non_primary.8b3ca4cd-e375-4c1e-ad73-163e99b4c960.0fd4ba93-e753-43f0-bd54-1d4e72a04239 147 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:122 copied) ########################################## csi-vol-e0d50304-b811-4c55-b6d4-570dbb874dc4 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 129 .mirror.non_primary.6f24657c-4ba8-43d9-99d6-3368632c6598.edf0aeb3-23a3-4f0f-8d13-cbf2856ec938 77 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:118 copied) ########################################## csi-vol-ee765cc3-119c-4c49-b457-d9335e235a42 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 130 .mirror.non_primary.7c0489e9-9767-4e64-b0f2-b04e041d36dc.fdbb8189-a81d-4c26-b174-fdd162bd4041 44 GiB Fri Oct 20 09:25:03 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:123 copied) ########################################## csi-vol-f58f166f-53e4-448e-8cb2-985502b2bb89 SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 119 .mirror.non_primary.7f007c71-a25a-45ed-96f6-a99ee2104593.8072f62b-c6f8-4859-aec9-e25324cbe9ed 58 GiB Fri Oct 20 09:25:01 2023 mirror (non-primary peer_uuids:[] 4079ccab-4468-49a8-9efd-c66445560eed:99 copied) ########################################## On hub ------ $ oc get placement -A NAMESPACE NAME SUCCEEDED REASON SELECTEDCLUSTERS openshift-gitops all-openshift-clusters True AllDecisionsScheduled 2 openshift-gitops busybox-1-placement True AllDecisionsScheduled 1 (venv) keerthana_manoharan:~$ oc get applications -A NAMESPACE NAME SYNC STATUS HEALTH STATUS openshift-gitops busybox-1-kmanohar-clu1 Synced Healthy (venv) keerthana_manoharan:~$ oc get placementrule -A NAMESPACE NAME AGE REPLICAS openshift-operators plrule-0eb4d5937ba374896ab0a0f1f8d824acedbecc4 146m openshift-operators plrule-b6a6f041c1181265bbc16e669ce8d5201dbd902 146m oc get drpc NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE busybox-1-placement-drpc 92m kmanohar-clu1 Deployed oc get drpc busybox-1-placement-drpc -o yaml | grep lastGroup lastGroupSyncBytes: 38322176 lastGroupSyncDuration: 1s lastGroupSyncTime: "2023-10-20T09:35:01Z" ___________________________________________________________- After the deletion of the application ===================================== On C1 ----- $ rbd mirror snapshot schedule status SCHEDULE TIME IMAGE $ rbd mirror snapshot schedule list --recursive POOL NAMESPACE IMAGE SCHEDULE $ for i in $(rbd ls -p ocs-storagecluster-cephblockpool); do echo $i; rbd snap ls ocs-storagecluster-cephblockpool/$i --all 2>/dev/null; echo "##########################################";done csi-vol-0a9ddb11-5c8c-411e-abb1-b4aa4ca2be68 ########################################## sh-5.1$ rbd mirror pool status ocs-storagecluster-cephblockpool health: OK daemon health: OK image health: OK images: 0 total $ pods No resources found in appset-busybox-1-c1 namespace. $ pvc No resources found in appset-busybox-1-c1 namespace. $ oc get vr,vrg No resources found in appset-busybox-1-c1 namespace. On C2 ----- $ for i in $(rbd ls -p ocs-storagecluster-cephblockpool); do echo $i; rbd snap ls ocs-storagecluster-cephblockpool/$i --all 2>/dev/null; echo "##########################################";done csi-vol-78a5f290-5d7f-4f95-ac62-7370f64b9770 ########################################## $ rbd mirror pool status ocs-storagecluster-cephblockpool health: OK daemon health: OK image health: OK images: 0 total On Hub ------ oc get drpc No resources found in openshift-gitops namespace. $ oc get placement -A NAMESPACE NAME SUCCEEDED REASON SELECTEDCLUSTERS openshift-gitops all-openshift-clusters True AllDecisionsScheduled 2 $ oc get applications -A No resources found $ oc get placementrule -A NAMESPACE NAME AGE REPLICAS openshift-operators plrule-0eb4d5937ba374896ab0a0f1f8d824acedbecc4 3h24m openshift-operators plrule-b6a6f041c1181265bbc16e669ce8d5201dbd902 3h24m Additional Info : ---------------- Tried the operation multiple times with multiple workloads. Verified on : ------------ OCP - 4.14.0-0.nightly-2023-10-17-113123 ODF - 4.14.0-150 MCO - 4.14.0-150 Submariner - 0.16.0 (brew.registry.redhat.io/rh-osbs/iib:599799) ACM - 2.9.0-2.9.0-DOWNSTREAM-2023-10-03-20-08-35 Ceph Version - ceph version 17.2.6-146.el9cp (1d01c2b30b5fd39787bb8804707c4b2e52e30137) quincy (stable) Must Gather - ----------- c1 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/bz-v/bz-2239140/c1/ c2 - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/bz-v/bz-2239140/c2/ Hub - http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/keerthana/bz-v/bz-2239140/hub/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.14.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6832