Description of problem (please be detailed as possible and provide log snippests): Deleting a storageconsumer which is missing heartbeat from client does not automatically deletes the cephblockpoolradosnamespace where rbd images is present. There was no subvolumes present. Cephfilesystemsubvolumegroups are deleted. rook-ceph-operator logs: 2024-05-17 08:43:58.757103 I | blockpool-rados-namespace-controller: deleting ceph blockpool rados namespace object "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" 2024-05-17 08:43:58.829325 I | blockpool-rados-namespace-controller: starting cleanup of the ceph resources for radosNamesapce "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" in namespace "openshift-storage" 2024-05-17 08:43:58.829356 I | op-k8sutil: format and nodeName longer than 53 chars, nodeName ocs-storagecluster-cephblockpool-cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7 will be d825e5f7e46288358a2152b33b7eb552 2024-05-17 08:43:58.836240 E | blockpool-rados-namespace-controller: failed to reconcile "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" failed to create clean up job for ceph blockpool rados namespace "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7": failed to run clean up job to clean the ceph resources in radosNamesapce "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7": failed to run clean up job for "CephBlockPoolRadosNamespace" resource named "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" in namespace "openshift-storage": jobs.batch "cleanup-radosnamespace-d825e5f7e46288358a2152b33b7eb552" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil> Logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/ibm-baremetal2/ibm-baremetal2_20240429T123517/logs/testcases_1715936596/ibm-baremetal2/ Storageclient name was storage-client-provider-bm2 . This was created on another cluster. Cluster id was 907d63de-0f0c-4fd7-b785-b6af39b2ad5b. Deleted storageconsumer name was storageconsumer-907d63de-0f0c-4fd7-b785-b6af39b2ad5b. Heartbeat missing since 3 days because the client cluster was deleted. Storageconsumer storageconsumer-907d63de-0f0c-4fd7-b785-b6af39b2ad5b deleted from UI. Result: * Storagerequests deleted. * Cephfilesystemsubvolumegroups “cephfilesystemsubvolumegroup-ac21295f28054bc15e57157ba7c73ce2” and “cephfilesystemsubvolumegroup-f9899912fb09aa038dc76d75989265cd” deleted. There was no subvolume present under both cephfilesystemsubvolumegroup. * There were 2 cephblockpoolradosnamespace present. cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7 did not delete because it is having a an rbd image. $ rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7 NAME SIZE PARENT FMT PROT LOCK csi-vol-97044c10-dd2b-4454-b477-999c81f1b539 1 GiB 2 cephradosnamespace-270c8f5f52d52f177b7a58b62add6a78 deleted which did not have an rbd image. ======================================================= Version of all relevant components (if applicable): % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.0-ec.5 True False 16d Cluster version is 4.16.0-ec.5 % oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.16.0-96.stable NooBaa Operator 4.16.0-96.stable mcg-operator.v4.16.0-92.stable Succeeded ocs-client-operator.v4.16.0-96.stable OpenShift Data Foundation Client 4.16.0-96.stable ocs-client-operator.v4.16.0-92.stable Succeeded ocs-operator.v4.16.0-96.stable OpenShift Container Storage 4.16.0-96.stable ocs-operator.v4.16.0-92.stable Succeeded odf-csi-addons-operator.v4.16.0-96.stable CSI Addons 4.16.0-96.stable odf-csi-addons-operator.v4.16.0-92.stable Succeeded odf-operator.v4.16.0-96.stable OpenShift Data Foundation 4.16.0-96.stable odf-operator.v4.16.0-92.stable Succeeded odf-prometheus-operator.v4.16.0-96.stable Prometheus Operator 4.16.0-96.stable odf-prometheus-operator.v4.16.0-92.stable Succeeded recipe.v4.16.0-96.stable Recipe 4.16.0-96.stable Succeeded rook-ceph-operator.v4.16.0-96.stable Rook-Ceph 4.16.0-96.stable rook-ceph-operator.v4.16.0-92.stable Succeeded The cluster which is having the storageclient also have the same OCP version. Client operator is 4.16.0-85.stable. ========================================== Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Leftovers that consume storage. Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 3 Can this issue reproducible? Reporting the fist instance Can this issue reproduce from the UI? Storageconsumer was deleted from UI =============================== If this is a regression, please provide more details to justify this: Steps to Reproduce: Follow the steps 1-4 given in https://bugzilla.redhat.com/show_bug.cgi?id=2276055#c8 under "for 3" ================================ Actual results: cephblockpoolradosnamespace and RBD image are not removed. Expected results: cephblockpoolradosnamespace, Cephfilesystemsubvolumegroups, RBD image and subvolume associated with the storageconsumer should be removed. Additional info:
I have checked the scenario with client deletion from UI (Storage / Storage Clients / trash icon ) on Provider cluster with odf-operator.v4.16.0-108.stable and Client cluster ocs-client-operator.v4.16.0-108.stable The issue is not resolved. Steps: c get storageconsumer -A -o jsonpath='{range .items[*]}{.metadata.name} {.status.client.clusterId}{"\n"}{end}' storageconsumer-4f974f42-2301-460a-90a5-2607cabea062 4f974f42-2301-460a-90a5-2607cabea062 storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1 902cd8c9-4115-4424-b8d7-1cf4127135d1 storageconsumer-bcff8cf4-a660-4c2d-8834-051fcb373021 bcff8cf4-a660-4c2d-8834-051fcb373021 storageconsumer-bd1ac952-86c2-4d0e-a6b2-92020b41e959 bd1ac952-86c2-4d0e-a6b2-92020b41e959 storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f c01582a4-80f7-4aac-bf83-002fa086394f storageconsumer-e5766b2d-e770-456f-8c9f-cc84140f8455 e5766b2d-e770-456f-8c9f-cc84140f8455 compare with storageclient CR - they dont refer one another id: f452a26f-2163-4c73-a2ee-f8495ddd882c step 1 switch to Client hcp415-bm3-i step 2 get clusterID oc get clusterversions.config.openshift.io version -n openshift-storage-client -o jsonpath='{.spec.clusterID}' 902cd8c9-4115-4424-b8d7-1cf4127135d1 step 3 oc get cronjob -n openshift-storage-client oc patch cronjob storageclient-e5fb1f06bee2517f-status-reporter -n openshift-storage-client -p '{"spec":{"suspend":true}}' set timer step 4 switch back to Provider step 5 get associated storagerequests on Provider: oc get storagerequest -A -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' openshift-storage storagerequest-1df0252fd0b9919690f9f97306a25b47 openshift-storage storagerequest-92c3e721d8d5240b525c08a20d466ab3 step 6 get associated cephradosnamespace oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}' cephradosnamespace-db0cd87b2701f9e25f84de4254f8911e% step 7 get associated cephfilesystemsubvolumegroups oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}' cephfilesystemsubvolumegroup-d1926f84e3f12c97361f29e8e431bcee step 8 after 5 min check that hearbeat stopped and last heartbeat updated 5 min ago, alert appeared go to UI of Provider and delete client with clusterID saved in step 2 902cd8c9-4115-4424-b8d7-1cf4127135d1 screen recording https://drive.google.com/file/d/1AowHjJi7Il5_j_BvNHA3VOTv_pEneCtF/view?usp=sharing
I have tested the same with another client when subvolumes and rbd images exist. Both provider and client are ocs-client-operator.v4.16.0-108.stable (interpreter) ➜ ocs-ci git:(master) ✗ oc get storagerequest -A -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' openshift-storage storagerequest-3da25b2c25c01bf68207bef8a66c6e27 openshift-storage storagerequest-9019dcf8665ea8ddc2187c82230b0b50 openshift-storage storagerequest-c78afdf62730a457f004413c0e308012 (interpreter) ➜ ocs-ci git:(master) ✗ oc get storageconsumer storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f -n openshift-storage Error from server (NotFound): storageconsumers.ocs.openshift.io "storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f" not found (interpreter) ➜ ocs-ci git:(master) ✗ oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}' cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 cephradosnamespace-86f0f753d45ee87382a8678d036d98ca% (interpreter) ➜ ocs-ci git:(master) ✗ oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}' \cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc% (interpreter) ➜ ocs-ci git:(master) ✗ (interpreter) ➜ ocs-ci git:(master) ✗ (interpreter) ➜ ocs-ci git:(master) ✗ tool zsh: command not found: tool (interpreter) ➜ ocs-ci git:(master) ✗ toolbash ocsinitialization.ocs.openshift.io/ocsinit patched sh-5.1$ ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc [ { "name": "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" }, { "name": "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b" } ] sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df sh: csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df: command not found sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df screen recording - https://drive.google.com/file/d/1LMkmOCLK7_dhUGlKhEglMm-0RRDpChqi/view?usp=sharing
(In reply to Daniel Osypenko from comment #13) > I have tested the same with another client when subvolumes and rbd images > exist. Both provider and client are ocs-client-operator.v4.16.0-108.stable Please provide the cluster where this is happening.
https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/38138/ copied to dm
(In reply to Daniel Osypenko from comment #15) > https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe- > deploy-ocs-cluster/38138/ > copied to dm Thanks Daniel Checked the cluster. I see that the resources mentioned in the comment are deleted and the clean up jobs are working correctly. CephSubVolumeGroup: `cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc` ❯ oc exec -it rook-ceph-tools-555d6cd8f7-7v529 sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. sh-5.1$ ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc Error ENOENT: subvolume group 'cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc' does not exist sh-5.1$ exit exit Cleanup job logs: ``` ❯ oc logs cleanup-svg-86529c108be12e0cc51fef8a3c8cc989-rjg7x 2024/05/30 15:12:33 maxprocs: Leaving GOMAXPROCS=64: CPU quota undefined 2024-05-30 15:12:33.243205 I | rookcmd: starting Rook v4.16.0-0.9b2caf9cc037c2396b89ae2116a98795b6acd978 with arguments '/usr/local/bin/rook ceph clean CephFilesystemSubVolumeGroup' 2024-05-30 15:12:33.243329 I | rookcmd: flag values: --help=false, --log-level=DEBUG 2024-05-30 15:12:33.243917 I | cleanup: starting clean up cephFS subVolumeGroup resource "cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc" 2024-05-30 15:12:33.243934 D | exec: Running command: ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json 2024-05-30 15:12:33.586566 I | cleanup: starting clean up of subvolume "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" 2024-05-30 15:12:33.586586 I | cleanup: OMAP value for the object "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" is "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" 2024-05-30 15:12:33.586607 D | exec: Running command: rados getomapval csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef csi.volname -p ocs-storagecluster-cephfilesystem-metadata --namespace csi /dev/stdout --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:33.626217 I | cleanup: OMAP key for the OIMAP value "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" is "ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8" 2024-05-30 15:12:33.626237 D | exec: Running command: rados rm csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:33.665008 I | cephclient: successfully deleted omap value "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" for pool "ocs-storagecluster-cephfilesystem-metadata" 2024-05-30 15:12:33.665045 D | exec: Running command: rados rmomapkey csi.volumes.default ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8 -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:33.702992 I | cephclient: successfully deleted omap key "ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8" for pool "ocs-storagecluster-cephfilesystem-metadata" 2024-05-30 15:12:33.703012 D | exec: Running command: ceph fs subvolume snapshot ls ocs-storagecluster-cephfilesystem csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef --group_name cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json 2024-05-30 15:12:34.032427 D | exec: Running command: ceph fs subvolume rm ocs-storagecluster-cephfilesystem csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --force --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json 2024-05-30 15:12:34.347869 I | cleanup: starting clean up of subvolume "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b" 2024-05-30 15:12:34.347889 I | cleanup: OMAP value for the object "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b" is "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b" 2024-05-30 15:12:34.347904 D | exec: Running command: rados getomapval csi.volume.41f55c47-0854-4abb-bd99-46f10301499b csi.volname -p ocs-storagecluster-cephfilesystem-metadata --namespace csi /dev/stdout --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:34.386386 I | cleanup: OMAP key for the OIMAP value "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b" is "ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f" 2024-05-30 15:12:34.386418 D | exec: Running command: rados rm csi.volume.41f55c47-0854-4abb-bd99-46f10301499b -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:34.424939 I | cephclient: successfully deleted omap value "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b" for pool "ocs-storagecluster-cephfilesystem-metadata" 2024-05-30 15:12:34.424974 D | exec: Running command: rados rmomapkey csi.volumes.default ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring 2024-05-30 15:12:34.464664 I | cephclient: successfully deleted omap key "ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f" for pool "ocs-storagecluster-cephfilesystem-metadata" 2024-05-30 15:12:34.464698 D | exec: Running command: ceph fs subvolume snapshot ls ocs-storagecluster-cephfilesystem csi-vol-41f55c47-0854-4abb-bd99-46f10301499b --group_name cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json 2024-05-30 15:12:34.816279 D | exec: Running command: ceph fs subvolume rm ocs-storagecluster-cephfilesystem csi-vol-41f55c47-0854-4abb-bd99-46f10301499b cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --force --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json 2024-05-30 15:12:35.151179 I | cleanup: successfully cleaned up cephFS subVolumeGroup "cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc" ``` -------------------------------- CephRadosNamespace: sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 rbd: namespace 'cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6' does not exist. rbd: listing images failed: (2) No such file or directory sh-5.1$ ❯ oc get cephblockpoolradosnamespaces cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 -o yaml Error from server (NotFound): cephblockpoolradosnamespaces.ceph.rook.io "cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6" not found ---------------------- So moving this back to on_QA to verify again.
resources are left behind, please check it @sapillai ➜ oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}' cephradosnamespace-db0cd87b2701f9e25f84de4254f8911e% ➜ oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}' cephfilesystemsubvolumegroup-d1926f84e3f12c97361f29e8e431bcee% and ➜ oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}' cephradosnamespace-961a77489221c265904ce799b20e5bb2% ➜ oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}' cephfilesystemsubvolumegroup-7bff4824be367f35c0a05ba23a4ec6bf%
Verified in version: OCP 4.16.0-ec.6 ODF 4.16.0-110 Same version of client and OCP in the hosted cluster. Client created on hosted cluster using agent. Created two additional storageclaims on client side. This created new cephblockpoolradosnamespace and cephfilesystemsubvolumegroup. So there are two cephblockpoolradosnamespaces and cephfilesystemsubvolumegroups associated with the storageconsumer. Created PVCs using all the 4 storageclasses. Created PVCs are RBD - Block and Filesystem volume mode. CephFS - Filesystem volume mode. 1 GB of data was present on each volume at the time of deleting the hosted cluster. Snapshot of the PVCs were present. Deleted the hosted cluster where the storageclient is present. After 20 minutes, deleted the storageclient from storage clients page in provider cluster UI. At the time of deletion, last heartbeat was showing as 20 minutes ago in the UI. Cephblockpoolradosnamespaces and cephfilesystemsubvolumegroups are deleted. Test details and outputs are given in https://bugzilla.redhat.com/show_bug.cgi?id=2280813#c6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:4591