Bug 2170051

Summary:	Used storage taken by CSI Subvolumes that are not linked to a PVC are not visible in openshift
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Solli Dennis <sdennis>
Component:	ceph-monitoring	Assignee:	Divyansh Kamboj <dkamboj>
Status:	ASSIGNED ---	QA Contact:	Harish NV Rao <hnallurv>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	bniver, dpandit, gfarnum, jpeyrard, muagarwa, odf-bz-bot, skatiyar, sostapov
Target Milestone:	---	Flags:	sheggodu: needinfo? (dpandit)
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Solli Dennis 2023-02-15 14:38:59 UTC

Description of problem (please be detailed as possible and provide log
snippests):

Sometimes orphan subvolumes are created in cephfs, by deleting a PVC in openshift but its associated CSI subvolume doesn't get deleted (e.g when filesystem is full and a pvc is deleted) .

When orphan subvolumes are present in CEPH fs, we can see the space they take with the command 'ceph df' . However, this is not reflected in Openshift, from the webconsole it looks like the space taken by the orphan subvolumes was released successfully when the associated pvc was deleted.

Version of all relevant components (if applicable):
openshift 4.8
ODF 4.7

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Since there is no alert to the customer that orphan subvolumes are present in the CEPHfs, and the space taken by those csi subvolumes is not reflected in openshift. Customer will dimension their workload based on what's reported by openshift which leads to an unexpected filesystem full scenario.

Is there any workaround available to the best of your knowledge?

We created a shell script that provides a list of the orphan subvolumes present in the cluster, then we used said list to delete them.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

Can this issue reproducible?

Yes. The original issue was caused by the customer deleting PVCs while the CEPHfs were blocked with the alarm filesystem full. However, we were able to reproduce it by following the steps below:

1.- create a dummy deployment with a mounted a volume using CEPHfs
2.- change the ReclaimPolicy of the PV to 'Retain'
3.- delete the pv/pvc

As a result, the PV is deleted, but the associate csi subvolume remains in Cephfs

Can this issue reproduce from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. create a dummy deployment with an mounted a volume using CEPHfs
2. change the ReclaimPolicy of the PV to 'Retain'
3. delete the pv/pvc

Actual results:

The PV is deleted, but the associate csi subvolume remains in Cephfs.

Openshift webconsole reports the used space by the PV has been released, however the space remains in used in the cephfs.

Expected results:

I'd expected the following:

1.- Openshift to have some type of alarm to indicate there are orphan subvolumes present in cephFS.
2.- Openshift to report the same usage of storage as CEPH.

Additional info:

Comment 6 Sanjal Katiyar 2023-04-11 06:30:43 UTC

moving to 4.14 as we don;t have enough time left to take it up in 4.13 release...

Comment 7 Sanjal Katiyar 2023-04-26 08:50:54 UTC

Being discussed currently: https://chat.google.com/room/AAAABTq-uCM/y0lVSUmh84U

Comment 8 Sanjal Katiyar 2023-04-28 08:26:48 UTC

(In reply to Sanjal Katiyar from comment #7)
> Being discussed currently:
> https://chat.google.com/room/AAAABTq-uCM/y0lVSUmh84U

we have 2 parts to UI:
1. Raw capacity: which displays absolute total disk, uses "ceph_cluster_total_bytes" and "ceph_cluster_total_used_raw_bytes". If Ceph itself is not reporting the "orphan subvolumes" via these metrics, it can't be reflected on the UI as well.

2. Used capacity breakdown: which displays logical consumption, uses "kubelet_volume_stats_used_bytes" which would mean it can;t report information related to PVs with are not mounted (or "Retain"-ed subvolumes). Reason for using so, I believe would be, that we don;t have any metric for block+file as of yet to display logical consumption any other way.
We do have an epic to make "Used capacity breakdown" card more robust: https://issues.redhat.com/browse/RHSTOR-3425

Moving this to monitoring team for now.