Bug 2280813
| Summary: | Cephblockpoolradosnamespace, cephfilesystemsubvolumegroup and storagerequest are not deleted with storageconsumer deletion | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Jilju Joy <jijoy> |
| Component: | ocs-operator | Assignee: | Leela Venkaiah Gangavarapu <lgangava> |
| Status: | CLOSED ERRATA | QA Contact: | Jilju Joy <jijoy> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.16 | CC: | lgangava, nigoyal, odf-bz-bot |
| Target Milestone: | --- | ||
| Target Release: | ODF 4.16.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | isf-provider | ||
| Fixed In Version: | 4.16.0-110 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-07-17 13:22:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Verified in version:
OCP 4.16.0-ec.6
ODF 4.16.0-110
Same version of client and OCP in the hosted cluster.
Client created on hosted cluster using agent. Created two additional storageclaims on client side. This created new cephblockpoolradosnamespace and cephfilesystemsubvolumegroup. So there are two cephblockpoolradosnamespaces and cephfilesystemsubvolumegroups associated with the storageconsumer. Created PVCs using all the 4 storageclasses. Created PVCs are RBD - Block and Filesystem volume mode. CephFS - Filesystem volume mode. 1 GB of data was present on each volume at the time of deleting the hosted cluster. Snapshot of the PVCs were present.
Storageclient is
% oc get storageclient
NAME PHASE CONSUMER
ocs-storagecluster Connected 4f6ff6c5-cbbe-4ba0-a050-e72c694c8a52
Storageconsumer on provider is
% oc get storageconsumer storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449
NAME AGE
storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449 6d14h
% oc get storageconsumer storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449 -o yaml
apiVersion: ocs.openshift.io/v1alpha1
kind: StorageConsumer
metadata:
annotations:
ocs.openshift.io/provider-onboarding-ticket: eyJpZCI6ImU0ZmEzZjAxLWMyMjctNGFiNi1hY2Y2LWNmNmI3ZGZkNDg3YSIsImV4cGlyYXRpb25EYXRlIjoiMTcxNzE3NTMzMSJ9.JxueAyQ3DNmcWL8KaQsWWdUAHFUo/M2rcmOzRhzk/J2uK7CCPHMGlUt5+/Tp+tyq6jM0h628fWXO88ZNPIOON0QJKfF3Btw9FxLe1bOSWBfIpvIT4AcPfKJTWJcH2xVPM7QV6VghBoTDzA7oZkgxlrWeRyqPDP8GLbPnIrr1UR0FgfZ5i4ZdYSjk2tHy8v/n1lC7WqRvpM+OoKG8c0NvPNF0t/wBqfgFNy6Nh62CbAl7HLiJn+MpguhDZkBWLd+v+GfwfwUXNDr/MhGEnECnp483cIHDdQ5rLgOqSb8F7Nhd4GppKJgedPPKqEieyHXyDK9S65jOckY48z4g0FGgbf+g6MyTwZD0wPXtm8i6Y90xX5EPRuxrpYglAjqMT1jxYc4VJK0R5ZS1C1Ld5ZohM/KcTHVOa9KYOnxvB4dkCFIa6roQj+Zzn5nXXhrIE80Wu3iBsJvgNI4QASfHf2/P24xFrnRKztSi6XcbQKTkc44St8r9n45GMoEPnolD5vxY1b2UD7jA7ozbyKBOXocNv01Q4W1dmcyCvVGBNlRo9e6v32ORuQFhFGbUQv1EMFJBjozeWEK5bs86/wmsmJnv8m9hooCwl9LpcFG+uwHmyeZ5ll8T50vdHLdJoZxsOL1mDUdaXZYDVH4yOSXoQQC8VfSCrW7slteOfX08yvHpUXU=
creationTimestamp: "2024-05-29T17:09:30Z"
generation: 2
name: storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449
namespace: openshift-storage
resourceVersion: "55593964"
uid: 4f6ff6c5-cbbe-4ba0-a050-e72c694c8a52
spec:
enable: true
status:
cephResources:
- kind: CephClient
name: a850871d1f8bfe7b84af4dc61f335b6d
status: Ready
client:
clusterId: b6628a92-4780-4f93-a94b-adb7024dc449
clusterName: ""
name: ocs-storagecluster
operatorVersion: 4.16.0-110.stable
platformVersion: 4.16.0-ec.6
lastHeartbeat: "2024-06-05T07:45:15Z"
state: Ready
Storage requests - Created additional storage claim from client side
storagerequest-c6c7b03b46ffc2c135dfb3960ae8c740 - sharedfile
storagerequest-cd23231e044de2f2775320bbcf508999 - block
storagerequest-be9ad34a8be5c58875b1fd63fce10798 - additional ‘sharedfile’
storagerequest-0201bdac5d84f1b480a0581e0c308727 - additional ‘block’
Subvolumegroups
1. cephfilesystemsubvolumegroup-6a395559002d3087b38215eb641637ab
% oc rsh rook-ceph-tools-f6958bb97-f5nrj ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-6a395559002d3087b38215eb641637ab
[
{
"name": "csi-vol-7ad367ec-4956-4583-b88e-d10f4826ecc9"
},
{
"name": "csi-vol-942a89b8-3b57-4b7e-984d-ce81a53bbc16"
},
{
"name": "csi-vol-3d49d1f2-8e77-4d76-8fd3-1588d0078103"
},
{
"name": "csi-vol-f984dd06-b1f7-4e90-a7b4-402203ff2840"
},
{
"name": "csi-vol-8b5d464e-7017-48e6-a040-12bc7ef322c4"
},
{
"name": "csi-vol-e9e26adc-a74a-4c13-8614-db8b60e1f122"
}
]
2. cephfilesystemsubvolumegroup-e5ce334603fe1b1ea7eb913969ec3399
% oc rsh rook-ceph-tools-f6958bb97-f5nrj ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-e5ce334603fe1b1ea7eb913969ec3399
[
{
"name": "csi-vol-d2d710cf-9133-416d-81ef-88617b304533"
},
{
"name": "csi-vol-97c4a7cf-7fa2-4d8f-abf7-55bedab1cdf4"
},
{
"name": "csi-vol-d23576a6-767a-49fe-aedb-2ed56992a301"
},
{
"name": "csi-vol-cfffa01f-5c83-4229-ab13-54df4da05aac"
},
{
"name": "csi-vol-d515a7b1-7ff3-4ecf-8ae9-239bfb12d749"
},
{
"name": "csi-vol-f300b491-5ab8-4f65-a1d8-75c8e903ca1c"
},
{
"name": "csi-vol-206ee0fd-a542-46ff-bb22-43ce58c601f4"
},
{
"name": "csi-vol-d0ead75a-23f7-4a37-88ea-9c6728a76b85"
}
]
Radosnamespaces
2. cephradosnamespace-5ee26effcdd716c681549c517645d8f0
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-5ee26effcdd716c681549c517645d8f0
csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954
csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b
csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8
csi-vol-21e66cc2-40e9-42e3-b511-eefe39725ddf
csi-vol-33c68641-2cd3-4373-8831-acff19b1ca04
csi-vol-3452a7bf-2bd3-4d3b-8d9f-d955d48c00eb
csi-vol-423afeef-4e62-425d-a884-863688f6673a
csi-vol-5e572257-a507-4f0c-96d0-6a53ba678a27
csi-vol-64cd3738-6983-4c42-b8c0-ced3ed5e3d75
csi-vol-ae71c567-16ba-4219-851c-f1197d557eff
csi-vol-b7d6ac3f-d339-447e-aec8-79fef319a8fc
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-5ee26effcdd716c681549c517645d8f0
NAME SIZE PARENT FMT PROT LOCK
csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-64cd3738-6983-4c42-b8c0-ced3ed5e3d75@36f6299d-ec19-4023-9736-26c729349340 2
csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954@csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-64cd3738-6983-4c42-b8c0-ced3ed5e3d75@36f6299d-ec19-4023-9736-26c729349340 2
csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-33c68641-2cd3-4373-8831-acff19b1ca04@8f0dd594-f02e-44ae-b852-484dc43ec2f8 2
csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b@csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-33c68641-2cd3-4373-8831-acff19b1ca04@8f0dd594-f02e-44ae-b852-484dc43ec2f8 2
csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-21e66cc2-40e9-42e3-b511-eefe39725ddf@7e2b4223-f09f-466d-9541-19afa85e2796 2
csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8@csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-vol-21e66cc2-40e9-42e3-b511-eefe39725ddf@7e2b4223-f09f-466d-9541-19afa85e2796 2
csi-vol-21e66cc2-40e9-42e3-b511-eefe39725ddf 3 GiB 2
csi-vol-33c68641-2cd3-4373-8831-acff19b1ca04 3 GiB 2
csi-vol-3452a7bf-2bd3-4d3b-8d9f-d955d48c00eb 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954@csi-snap-28e7c4eb-c60d-4817-902c-f81f7fd29954 2
csi-vol-423afeef-4e62-425d-a884-863688f6673a 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b@csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b 2
csi-vol-5e572257-a507-4f0c-96d0-6a53ba678a27 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b@csi-snap-2a423828-1b68-4a28-b6cc-3b198f84420b 2
csi-vol-64cd3738-6983-4c42-b8c0-ced3ed5e3d75 3 GiB 2
csi-vol-ae71c567-16ba-4219-851c-f1197d557eff 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8@csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8 2
csi-vol-b7d6ac3f-d339-447e-aec8-79fef319a8fc 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-5ee26effcdd716c681549c517645d8f0/csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8@csi-snap-50888002-4dfd-4c40-9c80-2e1bc152eda8 2
2. cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -p ocs-storagecluster-cephblockpool --namespace cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20
csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c
csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6
csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5
csi-vol-22b37953-e61b-48e5-9d3d-10f393163fa0
csi-vol-23f7faea-c6c8-4a7e-a753-5afede8b0785
csi-vol-3dd19376-8cc3-4938-b0c3-91ef114bcec3
csi-vol-708b4f4b-9829-4d6c-b932-2bf277618e0a
csi-vol-7ec4511f-9220-428c-8385-4a8bad7e4aae
csi-vol-acc0dc3f-a988-475b-b330-d08d35e9e5c4
csi-vol-b143a421-7f27-4570-a2e3-13e8f9067d20
csi-vol-b5224027-c88a-49e8-9bfe-a64126d95856
csi-vol-ff1a48f8-2aef-4877-b405-a9273a1c0fef
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20
NAME SIZE PARENT FMT PROT LOCK
csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-b5224027-c88a-49e8-9bfe-a64126d95856@9e7f5122-f512-49f5-ac4d-744d2562dba6 2
csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c@csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-b5224027-c88a-49e8-9bfe-a64126d95856@9e7f5122-f512-49f5-ac4d-744d2562dba6 2
csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-ff1a48f8-2aef-4877-b405-a9273a1c0fef@0dfe253a-b800-4bff-83a5-28a8f3ac554d 2
csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6@csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-ff1a48f8-2aef-4877-b405-a9273a1c0fef@0dfe253a-b800-4bff-83a5-28a8f3ac554d 2
csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-22b37953-e61b-48e5-9d3d-10f393163fa0@72109d38-1bbe-46be-b55c-26eedcf4bee9 2
csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5@csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-vol-22b37953-e61b-48e5-9d3d-10f393163fa0@72109d38-1bbe-46be-b55c-26eedcf4bee9 2
csi-vol-22b37953-e61b-48e5-9d3d-10f393163fa0 3 GiB 2
csi-vol-23f7faea-c6c8-4a7e-a753-5afede8b0785 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5@csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5 2
csi-vol-3dd19376-8cc3-4938-b0c3-91ef114bcec3 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6@csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6 2
csi-vol-708b4f4b-9829-4d6c-b932-2bf277618e0a 3 GiB 2 excl
csi-vol-7ec4511f-9220-428c-8385-4a8bad7e4aae 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6@csi-snap-cd09cde4-a87e-4d2a-abb6-263c3f7cd7d6 2
csi-vol-acc0dc3f-a988-475b-b330-d08d35e9e5c4 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c@csi-snap-0d94abe9-e958-4ca0-8a53-d3d36b46a25c 2
csi-vol-b143a421-7f27-4570-a2e3-13e8f9067d20 3 GiB ocs-storagecluster-cephblockpool/cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20/csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5@csi-snap-d10722bc-34e2-478c-82a6-b8e515eeb3c5 2
csi-vol-b5224027-c88a-49e8-9bfe-a64126d95856 3 GiB 2
csi-vol-ff1a48f8-2aef-4877-b405-a9273a1c0fef 3 GiB 2
Deleted the hosted cluster where the storageclient is present. After 20 minutes, deleted the storageclient from storage clients page in provider cluster UI. At the time of deletion, last heartbeat was showing as 20 minutes ago in the UI.
Storageconsumer was removed quickly.
Storageconsumer is deleted:
% oc get storageconsumer storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449
Error from server (NotFound): storageconsumers.ocs.openshift.io "storageconsumer-b6628a92-4780-4f93-a94b-adb7024dc449" not found
Storagerequests deleted.
% oc get storagerequests storagerequest-c6c7b03b46ffc2c135dfb3960ae8c740 storagerequest-cd23231e044de2f2775320bbcf508999 storagerequest-be9ad34a8be5c58875b1fd63fce10798 storagerequest-0201bdac5d84f1b480a0581e0c308727
Error from server (NotFound): storagerequests.ocs.openshift.io "storagerequest-c6c7b03b46ffc2c135dfb3960ae8c740" not found
Error from server (NotFound): storagerequests.ocs.openshift.io "storagerequest-cd23231e044de2f2775320bbcf508999" not found
Error from server (NotFound): storagerequests.ocs.openshift.io "storagerequest-be9ad34a8be5c58875b1fd63fce10798" not found
Error from server (NotFound): storagerequests.ocs.openshift.io "storagerequest-0201bdac5d84f1b480a0581e0c308727" not found
Subvolumegroups deleted.
% oc rsh rook-ceph-tools-f6958bb97-f5nrj ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-6a395559002d3087b38215eb641637ab
Error ENOENT: subvolume group 'cephfilesystemsubvolumegroup-6a395559002d3087b38215eb641637ab' does not exist
command terminated with exit code 2
% oc rsh rook-ceph-tools-f6958bb97-f5nrj ceph fs subvolume ls ocs-storagecluster-cephfilesystem cephfilesystemsubvolumegroup-e5ce334603fe1b1ea7eb913969ec3399
Error ENOENT: subvolume group 'cephfilesystemsubvolumegroup-e5ce334603fe1b1ea7eb913969ec3399' does not exist
command terminated with exit code 2
Radosnamespaces deleted
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-5ee26effcdd716c681549c517645d8f0
rbd: namespace 'cephradosnamespace-5ee26effcdd716c681549c517645d8f0' does not exist.
rbd: listing images failed: (2) No such file or directory
command terminated with exit code 2
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd ls -l ocs-storagecluster-cephblockpool --namespace cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20
rbd: namespace 'cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20' does not exist.
rbd: listing images failed: (2) No such file or directory
command terminated with exit code 2
The radosnamespace cephradosnamespace-e3a6585f78f7f36c3202ebae61d96b20 and cephradosnamespace-5ee26effcdd716c681549c517645d8f0 are not listed.
% oc rsh rook-ceph-tools-f6958bb97-f5nrj rbd namespace list ocs-storagecluster-cephblockpool
NAME
cephradosnamespace-1518c2a5c832628b5140165c952bb2db
cephradosnamespace-279a78543c6d8f28fba43c9046d0ba77
cephradosnamespace-348fc3351f506121ec3444c7ac7f814a
cephradosnamespace-4075113d4b7cee60e638c64f77d35b2f
cephradosnamespace-4a4263fa3634feb0b74f8ecefb25fa3d
cephradosnamespace-5922ae1f367ecab41f8aadaa7c327ca9
cephradosnamespace-d0e0ee9087ba526a589f36e9a38959fa
cephradosnamespace-e09b8c7b6b2f387870fe943118e26219
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:4591 |
Description of problem (please be detailed as possible and provide log snippests): Deleting a storageconsumer which is missing heartbeat from client does not automatically deletes the associated storagerequest, cephblockpoolradosnamespace and cephfilesystemsubvolumegroup. Name of the storageclient from hosted cluster. % oc get storageclient NAME PHASE CONSUMER ocs-storagecluster Connected ac2ed050-d5ce-49cd-be39-343a6836eff0 % oc get storageclient ocs-storagecluster -n openshift-storage-client -o=jsonpath={.status.id} ac2ed050-d5ce-49cd-be39-343a6836eff0 On provider side, the storageconsumer name is storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 % oc get storageconsumer storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 -o yaml | grep ac2ed050-d5ce-49cd-be39-343a6836eff0 uid: ac2ed050-d5ce-49cd-be39-343a6836eff0 Cephfilesystemsubvolumegroups is cephfilesystemsubvolumegroup-822d83fa5a6f6ca4cb3dc8502a32bbae % oc get cephfilesystemsubvolumegroups cephfilesystemsubvolumegroup-822d83fa5a6f6ca4cb3dc8502a32bbae -o yaml | grep storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 ocs.openshift.io/storageconsumer-name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 Cephblockpoolradosnamespace is cephradosnamespace-d1e7af4edd1801d99d70245229722eae % oc get cephblockpoolradosnamespace cephradosnamespace-d1e7af4edd1801d99d70245229722eae -o yaml| grep storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 ocs.openshift.io/storageconsumer-name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 Cephblockpool is ocs-storagecluster-cephblockpool Cephfilesystem is ocs-storagecluster-cephfilesystem The storageconsumer is deleted from UI(Storage--> StorageClients page) % oc get storageconsumer storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 -n openshift-storage Error from server (NotFound): storageconsumers.ocs.openshift.io "storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304" not found cephfilesystemsubvolumegroups cephfilesystemsubvolumegroup-822d83fa5a6f6ca4cb3dc8502a32bbae is not deleted. % oc get cephfilesystemsubvolumegroups cephfilesystemsubvolumegroup-822d83fa5a6f6ca4cb3dc8502a32bbae -o yaml | grep storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 ocs.openshift.io/storageconsumer-name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 Cephblockpoolradosnamespace cephradosnamespace-d1e7af4edd1801d99d70245229722eae is not deleted. % oc get cephblockpoolradosnamespace cephradosnamespace-d1e7af4edd1801d99d70245229722eae -o yaml| grep storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 ocs.openshift.io/storageconsumer-name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 Storagerequests are not deleted % oc get storagerequest storagerequest-7fcd02b35d53d25c5f9377818ca4ba4c storagerequest-cbebe0ef42adfc73682d262fdf210ae7 NAME STORAGETYPE PHASE storagerequest-7fcd02b35d53d25c5f9377818ca4ba4c block Initializing storagerequest-cbebe0ef42adfc73682d262fdf210ae7 sharedfile Initializing % oc get storagerequest storagerequest-7fcd02b35d53d25c5f9377818ca4ba4c storagerequest-cbebe0ef42adfc73682d262fdf210ae7 -o yaml | grep storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 name: storageconsumer-276f92ad-8b8f-4aeb-beb5-5a9f6e0b8304 Logs http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/ibm-baremetal2/ibm-baremetal2_20240429T123517/logs/testcases_1715860519/ibm-baremetal2/ ======================================================================= Version of all relevant components (if applicable): % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.0-ec.5 True False 16d Cluster version is 4.16.0-ec.5 % oc get csv NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.16.0-96.stable NooBaa Operator 4.16.0-96.stable mcg-operator.v4.16.0-92.stable Succeeded ocs-client-operator.v4.16.0-96.stable OpenShift Data Foundation Client 4.16.0-96.stable ocs-client-operator.v4.16.0-92.stable Succeeded ocs-operator.v4.16.0-96.stable OpenShift Container Storage 4.16.0-96.stable ocs-operator.v4.16.0-92.stable Succeeded odf-csi-addons-operator.v4.16.0-96.stable CSI Addons 4.16.0-96.stable odf-csi-addons-operator.v4.16.0-92.stable Succeeded odf-operator.v4.16.0-96.stable OpenShift Data Foundation 4.16.0-96.stable odf-operator.v4.16.0-92.stable Succeeded odf-prometheus-operator.v4.16.0-96.stable Prometheus Operator 4.16.0-96.stable odf-prometheus-operator.v4.16.0-92.stable Succeeded recipe.v4.16.0-96.stable Recipe 4.16.0-96.stable Succeeded rook-ceph-operator.v4.16.0-96.stable Rook-Ceph 4.16.0-96.stable rook-ceph-operator.v4.16.0-92.stable Succeeded Hosted cluster which is having the storageclient also have the same OCP and client operator version. ======================================================================= Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Removing the storageconsumer does not do the cleanup correctly Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 3 Can this issue reproducible? Reporting the fist instance Can this issue reproduce from the UI? Storageconsumer was deleted from UI If this is a regression, please provide more details to justify this: Steps to Reproduce: Follow the steps 1-4 given in https://bugzilla.redhat.com/show_bug.cgi?id=2276055#c8 under "for 3" Actual results: storagerequest, cephblockpoolradosnamespace and cephfilesystemsubvolumegroup are not deleted with storageconsumer when the storageconsumer does not have the heartbeat from the storageclient. Expected results: storagerequest, cephblockpoolradosnamespace and cephfilesystemsubvolumegroup and any images or subvolumes should be removed Additional info: Provider is on BM. Hostedcluster is created using agent.