Bug 2280946 - Cephblockpoolradosnamespace and subvolumegroups not deleted with storageconsumer deletion
Summary: Cephblockpoolradosnamespace and subvolumegroups not deleted with storageconsu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: rook
Version: 4.16
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ODF 4.16.0
Assignee: Santosh Pillai
QA Contact: Jilju Joy
URL:
Whiteboard: isf-provider
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-05-17 09:46 UTC by Jilju Joy
Modified: 2024-07-17 13:23 UTC (History)
5 users (show)

Fixed In Version: 4.16.0-106
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-07-17 13:23:10 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github red-hat-storage rook pull 652 0 None open Bug 2280946: core: remove owner refs in resource cleanup jobs 2024-05-20 04:42:41 UTC
Github rook rook pull 14234 0 None open core: remove owner refs in resource cleanup jobs 2024-05-17 13:29:55 UTC
Red Hat Product Errata RHSA-2024:4591 0 None None None 2024-07-17 13:23:13 UTC

Description Jilju Joy 2024-05-17 09:46:54 UTC
Description of problem (please be detailed as possible and provide log
snippests):

Deleting a storageconsumer which is missing heartbeat from client does not automatically deletes the cephblockpoolradosnamespace where rbd images is present. There was no subvolumes present. Cephfilesystemsubvolumegroups are deleted.

rook-ceph-operator logs:

2024-05-17 08:43:58.757103 I | blockpool-rados-namespace-controller: deleting ceph blockpool rados namespace object "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7"
2024-05-17 08:43:58.829325 I | blockpool-rados-namespace-controller: starting cleanup of the ceph resources for radosNamesapce "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" in namespace "openshift-storage"
2024-05-17 08:43:58.829356 I | op-k8sutil: format and nodeName longer than 53 chars, nodeName ocs-storagecluster-cephblockpool-cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7 will be d825e5f7e46288358a2152b33b7eb552
2024-05-17 08:43:58.836240 E | blockpool-rados-namespace-controller: failed to reconcile "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" failed to create clean up job for ceph blockpool rados namespace "openshift-storage/cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7": failed to run clean up job to clean the ceph resources in radosNamesapce "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7": failed to run clean up job for "CephBlockPoolRadosNamespace" resource named "cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7" in namespace "openshift-storage": jobs.batch "cleanup-radosnamespace-d825e5f7e46288358a2152b33b7eb552" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>



Logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/ibm-baremetal2/ibm-baremetal2_20240429T123517/logs/testcases_1715936596/ibm-baremetal2/

Storageclient name was storage-client-provider-bm2 . This was created on another cluster. Cluster id was 907d63de-0f0c-4fd7-b785-b6af39b2ad5b.
Deleted storageconsumer name was storageconsumer-907d63de-0f0c-4fd7-b785-b6af39b2ad5b. Heartbeat missing since 3 days because the client cluster was deleted. 


Storageconsumer storageconsumer-907d63de-0f0c-4fd7-b785-b6af39b2ad5b deleted from UI.

Result:

* Storagerequests deleted.
* Cephfilesystemsubvolumegroups “cephfilesystemsubvolumegroup-ac21295f28054bc15e57157ba7c73ce2” and “cephfilesystemsubvolumegroup-f9899912fb09aa038dc76d75989265cd” deleted. There was no subvolume present under both cephfilesystemsubvolumegroup.

* There were 2 cephblockpoolradosnamespace present. cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7 did not delete because it is having a an rbd image. 

$ rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-e8366b16de5d54579a6a3c7f01bd7aa7
NAME                                          SIZE   PARENT  FMT  PROT  LOCK
csi-vol-97044c10-dd2b-4454-b477-999c81f1b539  1 GiB            2    

cephradosnamespace-270c8f5f52d52f177b7a58b62add6a78 deleted which did not have an rbd image.



=======================================================

Version of all relevant components (if applicable):
% oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-ec.5   True        False         16d     Cluster version is 4.16.0-ec.5

% oc get csv
NAME                                        DISPLAY                            VERSION            REPLACES                                    PHASE
mcg-operator.v4.16.0-96.stable              NooBaa Operator                    4.16.0-96.stable   mcg-operator.v4.16.0-92.stable              Succeeded
ocs-client-operator.v4.16.0-96.stable       OpenShift Data Foundation Client   4.16.0-96.stable   ocs-client-operator.v4.16.0-92.stable       Succeeded
ocs-operator.v4.16.0-96.stable              OpenShift Container Storage        4.16.0-96.stable   ocs-operator.v4.16.0-92.stable              Succeeded
odf-csi-addons-operator.v4.16.0-96.stable   CSI Addons                         4.16.0-96.stable   odf-csi-addons-operator.v4.16.0-92.stable   Succeeded
odf-operator.v4.16.0-96.stable              OpenShift Data Foundation          4.16.0-96.stable   odf-operator.v4.16.0-92.stable              Succeeded
odf-prometheus-operator.v4.16.0-96.stable   Prometheus Operator                4.16.0-96.stable   odf-prometheus-operator.v4.16.0-92.stable   Succeeded
recipe.v4.16.0-96.stable                    Recipe                             4.16.0-96.stable                                               Succeeded
rook-ceph-operator.v4.16.0-96.stable        Rook-Ceph                          4.16.0-96.stable   rook-ceph-operator.v4.16.0-92.stable        Succeeded

The cluster which is having the storageclient also have the same OCP version.  Client operator is 4.16.0-85.stable. 

==========================================
Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Leftovers that consume storage.

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
3

Can this issue reproducible?
Reporting the fist instance

Can this issue reproduce from the UI?
Storageconsumer was deleted from UI

===============================
If this is a regression, please provide more details to justify this:


Steps to Reproduce:
Follow the steps 1-4 given in https://bugzilla.redhat.com/show_bug.cgi?id=2276055#c8 under "for 3"
================================


Actual results:
cephblockpoolradosnamespace and RBD image are not removed. 

Expected results:
cephblockpoolradosnamespace, Cephfilesystemsubvolumegroups, RBD image and subvolume associated with the storageconsumer should be removed.

Additional info:

Comment 12 Daniel Osypenko 2024-05-29 18:04:00 UTC
I have checked the scenario with client deletion from UI (Storage / Storage Clients / trash icon ) on Provider cluster with odf-operator.v4.16.0-108.stable and Client cluster ocs-client-operator.v4.16.0-108.stable
The issue is not resolved.
Steps:

c get storageconsumer -A -o jsonpath='{range .items[*]}{.metadata.name} {.status.client.clusterId}{"\n"}{end}' 
 
storageconsumer-4f974f42-2301-460a-90a5-2607cabea062 4f974f42-2301-460a-90a5-2607cabea062
storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1 902cd8c9-4115-4424-b8d7-1cf4127135d1
storageconsumer-bcff8cf4-a660-4c2d-8834-051fcb373021 bcff8cf4-a660-4c2d-8834-051fcb373021
storageconsumer-bd1ac952-86c2-4d0e-a6b2-92020b41e959 bd1ac952-86c2-4d0e-a6b2-92020b41e959
storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f c01582a4-80f7-4aac-bf83-002fa086394f
storageconsumer-e5766b2d-e770-456f-8c9f-cc84140f8455 e5766b2d-e770-456f-8c9f-cc84140f8455

compare with storageclient CR - they dont refer one another
 id: f452a26f-2163-4c73-a2ee-f8495ddd882c

step 1
switch to Client hcp415-bm3-i

step 2
get clusterID
 oc get clusterversions.config.openshift.io version -n openshift-storage-client -o jsonpath='{.spec.clusterID}' 
902cd8c9-4115-4424-b8d7-1cf4127135d1

step 3
oc get cronjob -n openshift-storage-client 

oc patch cronjob storageclient-e5fb1f06bee2517f-status-reporter -n openshift-storage-client -p '{"spec":{"suspend":true}}'

set timer

step 4
switch back to Provider

step 5
get associated storagerequests on Provider:

oc get storagerequest -A -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}'
openshift-storage storagerequest-1df0252fd0b9919690f9f97306a25b47
openshift-storage storagerequest-92c3e721d8d5240b525c08a20d466ab3

step 6
get associated cephradosnamespace 

oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}'
cephradosnamespace-db0cd87b2701f9e25f84de4254f8911e%

step 7
get associated cephfilesystemsubvolumegroups

oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}'
cephfilesystemsubvolumegroup-d1926f84e3f12c97361f29e8e431bcee


step 8
after 5 min check that hearbeat stopped and last heartbeat updated 5 min ago, alert appeared

go to UI of Provider and delete client with clusterID saved in step 2
902cd8c9-4115-4424-b8d7-1cf4127135d1

screen recording https://drive.google.com/file/d/1AowHjJi7Il5_j_BvNHA3VOTv_pEneCtF/view?usp=sharing

Comment 13 Daniel Osypenko 2024-05-30 15:42:11 UTC
I have tested the same with another client when subvolumes and rbd images exist. Both provider and client are ocs-client-operator.v4.16.0-108.stable


(interpreter) ➜  ocs-ci git:(master) ✗ oc get storagerequest -A -o jsonpath='{range .items[?(@.metadata.ownerReferences[0].name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")]}{.metadata.namespace} {.metadata.name}{"\n"}{end}'
openshift-storage storagerequest-3da25b2c25c01bf68207bef8a66c6e27
openshift-storage storagerequest-9019dcf8665ea8ddc2187c82230b0b50
openshift-storage storagerequest-c78afdf62730a457f004413c0e308012
(interpreter) ➜  ocs-ci git:(master) ✗ oc get storageconsumer storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f -n openshift-storage
Error from server (NotFound): storageconsumers.ocs.openshift.io "storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f" not found
(interpreter) ➜  ocs-ci git:(master) ✗ oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}'
cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 cephradosnamespace-86f0f753d45ee87382a8678d036d98ca%                                                                 
(interpreter) ➜  ocs-ci git:(master) ✗ oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}'
\cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc%                                                                                                          (interpreter) ➜  ocs-ci git:(master) ✗  
(interpreter) ➜  ocs-ci git:(master) ✗ 
(interpreter) ➜  ocs-ci git:(master) ✗ tool
zsh: command not found: tool
(interpreter) ➜  ocs-ci git:(master) ✗ toolbash
ocsinitialization.ocs.openshift.io/ocsinit patched
sh-5.1$ ceph fs subvolume ls ocs-storagecluster-cephfilesystem  cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc
[
    {
        "name": "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef"
    },
    {
        "name": "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b"
    }
]
sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6
csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df
csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df
sh: csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df: command not found
sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6
csi-vol-65257493-e43c-4210-9fee-b9b97a8cd0df

screen recording - https://drive.google.com/file/d/1LMkmOCLK7_dhUGlKhEglMm-0RRDpChqi/view?usp=sharing

Comment 14 Santosh Pillai 2024-05-31 06:31:58 UTC
(In reply to Daniel Osypenko from comment #13)
> I have tested the same with another client when subvolumes and rbd images
> exist. Both provider and client are ocs-client-operator.v4.16.0-108.stable


Please provide the cluster where this is happening.

Comment 16 Santosh Pillai 2024-05-31 07:03:33 UTC
(In reply to Daniel Osypenko from comment #15)
> https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-
> deploy-ocs-cluster/38138/
> copied to dm

Thanks Daniel

Checked the cluster. I see that the resources mentioned in the comment are deleted and the clean up jobs are working correctly. 

CephSubVolumeGroup: `cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc`

❯ oc exec -it rook-ceph-tools-555d6cd8f7-7v529 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-5.1$ ceph fs subvolume ls ocs-storagecluster-cephfilesystem  cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc
Error ENOENT: subvolume group 'cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc' does not exist
sh-5.1$ exit
exit

Cleanup job logs: 

```
❯ oc logs cleanup-svg-86529c108be12e0cc51fef8a3c8cc989-rjg7x
2024/05/30 15:12:33 maxprocs: Leaving GOMAXPROCS=64: CPU quota undefined
2024-05-30 15:12:33.243205 I | rookcmd: starting Rook v4.16.0-0.9b2caf9cc037c2396b89ae2116a98795b6acd978 with arguments '/usr/local/bin/rook ceph clean CephFilesystemSubVolumeGroup'
2024-05-30 15:12:33.243329 I | rookcmd: flag values: --help=false, --log-level=DEBUG
2024-05-30 15:12:33.243917 I | cleanup: starting clean up cephFS subVolumeGroup resource "cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc"
2024-05-30 15:12:33.243934 D | exec: Running command: ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json
2024-05-30 15:12:33.586566 I | cleanup: starting clean up of subvolume "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef"
2024-05-30 15:12:33.586586 I | cleanup: OMAP value for the object "csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" is "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef"
2024-05-30 15:12:33.586607 D | exec: Running command: rados getomapval csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef csi.volname -p ocs-storagecluster-cephfilesystem-metadata --namespace csi /dev/stdout --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:33.626217 I | cleanup: OMAP key for the OIMAP value "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" is "ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8"
2024-05-30 15:12:33.626237 D | exec: Running command: rados rm csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:33.665008 I | cephclient: successfully deleted omap value "csi.volume.090e5c49-15f4-4d06-ae6a-8a42ef0f17ef" for pool "ocs-storagecluster-cephfilesystem-metadata"
2024-05-30 15:12:33.665045 D | exec: Running command: rados rmomapkey csi.volumes.default ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8 -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:33.702992 I | cephclient: successfully deleted omap key "ceph.volume.pvc-1b2d6de6-3edb-427e-aea0-bc595f4837a8" for pool "ocs-storagecluster-cephfilesystem-metadata"
2024-05-30 15:12:33.703012 D | exec: Running command: ceph fs subvolume snapshot ls ocs-storagecluster-cephfilesystem csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef --group_name cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json
2024-05-30 15:12:34.032427 D | exec: Running command: ceph fs subvolume rm ocs-storagecluster-cephfilesystem csi-vol-090e5c49-15f4-4d06-ae6a-8a42ef0f17ef cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --force --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json
2024-05-30 15:12:34.347869 I | cleanup: starting clean up of subvolume "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b"
2024-05-30 15:12:34.347889 I | cleanup: OMAP value for the object "csi-vol-41f55c47-0854-4abb-bd99-46f10301499b" is "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b"
2024-05-30 15:12:34.347904 D | exec: Running command: rados getomapval csi.volume.41f55c47-0854-4abb-bd99-46f10301499b csi.volname -p ocs-storagecluster-cephfilesystem-metadata --namespace csi /dev/stdout --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:34.386386 I | cleanup: OMAP key for the OIMAP value "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b" is "ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f"
2024-05-30 15:12:34.386418 D | exec: Running command: rados rm csi.volume.41f55c47-0854-4abb-bd99-46f10301499b -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:34.424939 I | cephclient: successfully deleted omap value "csi.volume.41f55c47-0854-4abb-bd99-46f10301499b" for pool "ocs-storagecluster-cephfilesystem-metadata"
2024-05-30 15:12:34.424974 D | exec: Running command: rados rmomapkey csi.volumes.default ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f -p ocs-storagecluster-cephfilesystem-metadata --namespace csi --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring
2024-05-30 15:12:34.464664 I | cephclient: successfully deleted omap key "ceph.volume.pvc-e4d51546-ae4f-4df0-8e32-f8a96f3a602f" for pool "ocs-storagecluster-cephfilesystem-metadata"
2024-05-30 15:12:34.464698 D | exec: Running command: ceph fs subvolume snapshot ls ocs-storagecluster-cephfilesystem csi-vol-41f55c47-0854-4abb-bd99-46f10301499b --group_name cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json
2024-05-30 15:12:34.816279 D | exec: Running command: ceph fs subvolume rm ocs-storagecluster-cephfilesystem csi-vol-41f55c47-0854-4abb-bd99-46f10301499b cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc --force --connect-timeout=15 --cluster=openshift-storage --conf=/var/lib/rook/openshift-storage/openshift-storage.config --name=client.admin --keyring=/var/lib/rook/openshift-storage/client.admin.keyring --format json
2024-05-30 15:12:35.151179 I | cleanup: successfully cleaned up cephFS subVolumeGroup "cephfilesystemsubvolumegroup-c103cd65023f6e7bb20f101b1122c7fc"
```

--------------------------------

CephRadosNamespace:

sh-5.1$ rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6
rbd: namespace 'cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6' does not exist.
rbd: listing images failed: (2) No such file or directory
sh-5.1$ 


❯ oc get cephblockpoolradosnamespaces cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6 -o yaml
Error from server (NotFound): cephblockpoolradosnamespaces.ceph.rook.io "cephradosnamespace-60867fefece3a1865a2c3f08ae49d1b6" not found


----------------------
So moving this back to on_QA to verify again.

Comment 17 Daniel Osypenko 2024-05-31 07:34:11 UTC
resources are left behind, please check it @sapillai 

➜  oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}'
cephradosnamespace-db0cd87b2701f9e25f84de4254f8911e%                                                                                                         
➜  oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-902cd8c9-4115-4424-b8d7-1cf4127135d1")].metadata.name}'
cephfilesystemsubvolumegroup-d1926f84e3f12c97361f29e8e431bcee%  

and 

➜  oc get cephblockpoolradosnamespaces -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}'
cephradosnamespace-961a77489221c265904ce799b20e5bb2%                                                                                                         
➜ oc get cephfilesystemsubvolumegroups -n openshift-storage -o jsonpath='{.items[?(@.metadata.labels.ocs\.openshift\.io/storageconsumer-name=="storageconsumer-c01582a4-80f7-4aac-bf83-002fa086394f")].metadata.name}'
cephfilesystemsubvolumegroup-7bff4824be367f35c0a05ba23a4ec6bf%

Comment 19 Jilju Joy 2024-06-05 14:28:03 UTC
Verified in version:
OCP 4.16.0-ec.6
ODF 4.16.0-110

Same version of client and OCP in the hosted cluster.

Client created on hosted cluster using agent. Created two additional storageclaims on client side. This created new cephblockpoolradosnamespace and cephfilesystemsubvolumegroup. So there are two cephblockpoolradosnamespaces and cephfilesystemsubvolumegroups associated with the storageconsumer. Created PVCs using all the 4 storageclasses. Created PVCs are RBD - Block and Filesystem volume mode. CephFS - Filesystem volume mode. 1 GB of data was present on each volume at the time of deleting the hosted cluster. Snapshot of the PVCs were present.

Deleted the hosted cluster where the storageclient is present. After 20 minutes, deleted the storageclient from storage clients page in provider cluster UI. At the time of deletion, last heartbeat was showing as 20 minutes ago in the UI.

Cephblockpoolradosnamespaces and cephfilesystemsubvolumegroups are deleted. Test details and outputs are given in https://bugzilla.redhat.com/show_bug.cgi?id=2280813#c6

Comment 20 errata-xmlrpc 2024-07-17 13:23:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591


Note You need to log in before you can comment on or make changes to this bug.