Description of problem (please be detailed as possible and provide log snippests): Overprovision Level Policy Control with custom storageclass, The pv is not deleted although RECLAIMPOLICY is Delete This issue does not reconstructed on "ocs-storagecluster-ceph-rbd" and "ocs-storagecluster-cephfs" storage classes Version of all relevant components (if applicable): ODF Version: 4.11.0-137 OCP Version: 4.11.0-0.nightly-2022-08-17-152830 Provider: Vmware Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1.Create project with “openshift-quota” label $ cat ns_ocs-quota-sc-test.yaml apiVersion: v1 kind: Namespace metadata: name: ocs-quota-sc-test labels: openshift-quota: quota-sc-test 2.Create custom storageclass "sc-test-fs" $ oc get sc sc-test-fs NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE sc-test-fs openshift-storage.cephfs.csi.ceph.com Delete Immediate true 32s $ oc describe sc sc-test-fs Name: sc-test-fs IsDefaultClass: No Annotations: <none> Provisioner: openshift-storage.cephfs.csi.ceph.com Parameters: clusterID=openshift-storage,csi.storage.k8s.io/controller-expand-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/controller-expand-secret-namespace=openshift-storage,csi.storage.k8s.io/node-stage-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/node-stage-secret-namespace=openshift-storage,csi.storage.k8s.io/provisioner-secret-name=secret-test-cephfs-2828c2b94f1545fdb9eed,csi.storage.k8s.io/provisioner-secret-namespace=openshift-storage,fsName=ocs-storagecluster-cephfilesystem,pool=ocs-storagecluster-cephfilesystem-data0 AllowVolumeExpansion: True MountOptions: noexec ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none> 3. Add 'overprovisionControl' section to storagecluster yaml file: oc -n openshift-storage patch storagecluster ocs-storagecluster -n openshift-storage -p '{"spec": {"overprovisionControl": [{"capacity": "8Gi","storageClassName":"sc-test-fs", "quotaName": "quota-sc-test", "selector": {"labels": {"matchLabels": {"openshift-quota":"quota-sc-test"}}}}]}}' --type merge overprovisionControl: - capacity: 8Gi quotaName: quota-sc-test selector: labels: matchLabels: openshift-quota: quota-sc-test storageClassName: sc-test-fs $ oc get storageclusters.ocs.openshift.io NAME AGE PHASE EXTERNAL CREATED AT VERSION ocs-storagecluster 3d11h Ready 2022-08-18T08:20:56Z 4.11.0 4.Check quota: oc describe clusterresourcequota sc-test-fs-quota-sc-test Created: 10 hours ago Labels: <none> Annotations: <none> Namespace Selector: ["ocs-quota-sc-test"] Label Selector: openshift-quota=quota-sc-test AnnotationSelector: map[] Resource Used Hard -------- ---- ---- sc-test-fs.storageclass.storage.k8s.io/requests.storage 0 8Gi 5.Create new pvc on "ocs-quota-sc-test" project: apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-test-5e6a84be1a1a4751bf28be64ae18ed4 namespace: ocs-quota-sc-test spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: sc-test-fs $ oc get pvc -n ocs-quota-sc-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-test-5e6a84be1a1a4751bf28be64ae18ed4 Bound pvc-a05441a9-438a-4a00-bcdd-7590d6b9a3d8 5Gi RWO sc-test-fs 26s 6.Check quota: $ oc describe clusterresourcequota sc-test-fs-quota-sc-test Created: 10 hours ago Labels: <none> Annotations: <none> Namespace Selector: ["ocs-quota-sc-test"] Label Selector: openshift-quota=quota-sc-test AnnotationSelector: map[] Resource Used Hard -------- ---- ---- sc-test-fs.storageclass.storage.k8s.io/requests.storage 5Gi 8Gi 7.Attached nginx pod to pvc: apiVersion: v1 kind: Pod metadata: name: pod-test-cephfs-4a3351ca779049fc84b338f2 namespace: ocs-quota-sc-test spec: containers: - image: quay.io/ocsci/nginx:latest name: web-server volumeMounts: - mountPath: /var/lib/www/html name: mypvc volumes: - name: mypvc persistentVolumeClaim: claimName: pvc-test-5e6a84be1a1a4751bf28be64ae18ed4 readOnly: false $ oc get pods -n ocs-quota-sc-test -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-test-cephfs-4a3351ca779049fc84b338f2 1/1 Running 0 85s 10.128.2.46 compute-2 <none> <none> 9. Resize PVC to 6Gi: 10. Delete Pod: $ oc delete pod pod-test-cephfs-4a3351ca779049fc84b338f2 -n ocs-quota-sc-test pod "pod-test-cephfs-4a3351ca779049fc84b338f2" deleted [oviner@fedora ~]$ oc get pods -n ocs-quota-sc-test -o wide No resources found in ocs-quota-sc-test namespace. 11.Delete PVC: [oviner@fedora ~]$ oc delete pvc pvc-test-5e6a84be1a1a4751bf28be64ae18ed4 -n ocs-quota-sc-test persistentvolumeclaim "pvc-test-5e6a84be1a1a4751bf28be64ae18ed4" deleted 12.Check PV status: [PV is not deleted!!!] oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-883224cd-94ac-4170-ae42-ae22c89480c0 6Gi RWO Delete Released ocs-quota-sc-test/pvc-test-b37c78d44980496fb8a35517806bec7 sc-test-fs 54m Actual results: The pv stuck on Released state although RECLAIMPOLICY is Delete Expected results: The pv deleted Additional info:
Moving it to the CSI driver, Pls correct the component if it does not belong to CSI.
I tested it many times so I copied the wrong PV on step 12
$ oc describe pv pvc-89dd7af0-5222-4a01-8589-df07592859f6 Name: pvc-89dd7af0-5222-4a01-8589-df07592859f6 Labels: <none> Annotations: pv.kubernetes.io/provisioned-by: openshift-storage.rbd.csi.ceph.com volume.kubernetes.io/provisioner-deletion-secret-name: secret-test-rbd-ac1aae9ed67549cdb12cff63 volume.kubernetes.io/provisioner-deletion-secret-namespace: openshift-storage Finalizers: [kubernetes.io/pv-protection] StorageClass: sc-test-blk Status: Released Claim: ocs-quota-sc-test/pvc-test-e5a725b5edd1453d970f8363e31008c Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 1Gi Node Affinity: <none> Message: Source: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: openshift-storage.rbd.csi.ceph.com FSType: ext4 VolumeHandle: 0001-0011-openshift-storage-0000000000000001-f2df2165-21f3-11ed-be11-0a580a830012 ReadOnly: false VolumeAttributes: clusterID=openshift-storage imageFeatures=layering imageFormat=2 imageName=csi-vol-f2df2165-21f3-11ed-be11-0a580a830012 journalPool=ocs-storagecluster-cephblockpool pool=ocs-storagecluster-cephblockpool storage.kubernetes.io/csiProvisionerIdentity=1661096881837-8081-openshift-storage.rbd.csi.ceph.com Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning VolumeFailedDelete 31s (x8 over 94s) openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-8f9985c66-jg8bh_72034dac-d962-4321-bf40-d0fd623c12aa rpc error: code = InvalidArgument desc = provided secret is empty
I deleted the secret. oc -n openshift-storage delete Secret secret-test-rbd-ac1aae9ed67549cdb12cff63