Bug 2120121

Description of problem (please be detailed as possible and provide log
snippests):
Overprovision Level Policy Control with custom storageclass, The pv is not deleted although RECLAIMPOLICY is Delete

This issue does not reconstructed on   "ocs-storagecluster-ceph-rbd" and "ocs-storagecluster-cephfs" storage classes

Version of all relevant components (if applicable):
ODF Version: 4.11.0-137
OCP Version: 4.11.0-0.nightly-2022-08-17-152830
Provider: Vmware

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:

1.Create project with “openshift-quota” label
$ cat ns_ocs-quota-sc-test.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: ocs-quota-sc-test
  labels:
    openshift-quota: quota-sc-test
 
2.Create custom storageclass "sc-test-fs"
$ oc get sc sc-test-fs 
NAME         PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
sc-test-fs   openshift-storage.cephfs.csi.ceph.com   Delete          Immediate           true                   32s

$ oc describe sc sc-test-fs 
Name:                  sc-test-fs
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           openshift-storage.cephfs.csi.ceph.com
Parameters:            clusterID=openshift-storage,csi.storage.k8s.io/controller-expand-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/controller-expand-secret-namespace=openshift-storage,csi.storage.k8s.io/node-stage-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/node-stage-secret-namespace=openshift-storage,csi.storage.k8s.io/provisioner-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/provisioner-secret-namespace=openshift-storage,fsName=ocs-storagecluster-cephfilesystem,pool=ocs-storagecluster-cephfilesystem-data0
AllowVolumeExpansion:  True
MountOptions:
  noexec
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

3. Add 'overprovisionControl' section to storagecluster yaml file:
oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec": {"overprovisionControl": [{"capacity": "8Gi","storageClassName":"sc-test-fs", "quotaName": "quota-sc-test", "selector": {"labels": {"matchLabels": {"openshift-quota":"quota-sc-test"}}}}]}}' --type merge

    overprovisionControl:
    - capacity: 8Gi
      quotaName: quota-sc-test
      selector:
        labels:
          matchLabels:
            openshift-quota: quota-sc-test
      storageClassName: sc-test-fs

$ oc get storageclusters.ocs.openshift.io 
NAME                 AGE     PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   3d11h   Ready              2022-08-18T08:20:56Z   4.11.0

4.Check quota:
 oc describe clusterresourcequota sc-test-fs-quota-sc-test
Created:	10 hours ago
Labels:		<none>
Annotations:	<none>
Namespace Selector: ["ocs-quota-sc-test"]
Label Selector: openshift-quota=quota-sc-test
AnnotationSelector: map[]
Resource						Used	Hard
--------						----	----
sc-test-fs.storageclass.storage.k8s.io/requests.storage	0	8Gi

5.Create new pvc on "ocs-quota-sc-test" project:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-test-5e6a84be1a1a4751bf28be64ae18ed4
  namespace: ocs-quota-sc-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: sc-test-fs

$ oc get pvc -n ocs-quota-sc-test 
NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-test-5e6a84be1a1a4751bf28be64ae18ed4   Bound    pvc-a05441a9-438a-4a00-bcdd-7590d6b9a3d8   5Gi        RWO            sc-test-fs     26s

6.Check quota:
$ oc describe clusterresourcequota sc-test-fs-quota-sc-test
Created:	10 hours ago
Labels:		<none>
Annotations:	<none>
Namespace Selector: ["ocs-quota-sc-test"]
Label Selector: openshift-quota=quota-sc-test
AnnotationSelector: map[]
Resource						Used	Hard
--------						----	----
sc-test-fs.storageclass.storage.k8s.io/requests.storage	5Gi	8Gi


7.Attached nginx pod to pvc:
apiVersion: v1
kind: Pod
metadata:
  name: pod-test-cephfs-4a3351ca779049fc84b338f2
  namespace: ocs-quota-sc-test
spec:
  containers:
  - image: quay.io/ocsci/nginx:latest
    name: web-server
    volumeMounts:
    - mountPath: /var/lib/www/html
      name: mypvc
  volumes:
  - name: mypvc
    persistentVolumeClaim:
      claimName: pvc-test-5e6a84be1a1a4751bf28be64ae18ed4
      readOnly: false

$ oc get pods -n ocs-quota-sc-test -o wide
NAME                                       READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
pod-test-cephfs-4a3351ca779049fc84b338f2   1/1     Running   0          85s   10.128.2.46   compute-2   <none>           <none>

9. Resize PVC to 6Gi:

10. Delete Pod:
$ oc delete pod pod-test-cephfs-4a3351ca779049fc84b338f2 -n ocs-quota-sc-test 
pod "pod-test-cephfs-4a3351ca779049fc84b338f2" deleted
[oviner@fedora ~]$ oc get pods -n ocs-quota-sc-test -o wide
No resources found in ocs-quota-sc-test namespace.

11.Delete PVC:
[oviner@fedora ~]$ oc delete pvc pvc-test-5e6a84be1a1a4751bf28be64ae18ed4 -n ocs-quota-sc-test 
persistentvolumeclaim "pvc-test-5e6a84be1a1a4751bf28be64ae18ed4" deleted

12.Check PV status: [PV is not deleted!!!]
 oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                                            STORAGECLASS                  REASON   AGE
pvc-883224cd-94ac-4170-ae42-ae22c89480c0   6Gi        RWO            Delete           Released   ocs-quota-sc-test/pvc-test-b37c78d44980496fb8a35517806bec7       sc-test-fs                             54m


Actual results:
The pv stuck on Released state although RECLAIMPOLICY is Delete

Expected results:
The pv deleted

Additional info: