Description of problem: In a SNO cluster we're experiencing an issue where PVs will remain in "Released" state for a long time after the PVC bounded to the PV is deleted. Cleaning and deleting the PV with reclaim policy Delete can take seconds, minutes, or even hours. For some reason the PVCache is not being updated properly and then when this function[1] gets called here[2] the comparison[3] always skip the volume, we added some logs and these is what we've seen: The if was comparing "Available" vs "Released", and the pv status was "Released" when we ran "oc get pv <pvName>" so it's like the cache is not being updated properly or at least not timely. [1] https://github.com/openshift/local-storage-operator/blob/7dccd4a42798c44d43153d5f21e9980075e9d122/vendor/sigs.k8s.io/sig-storage-local-static-provisioner/pkg/deleter/deleter.go#L70 [2] https://github.com/openshift/local-storage-operator/blob/master/diskmaker/controllers/deleter/reconcile.go#L105 [3] https://github.com/openshift/local-storage-operator/blob/7dccd4a42798c44d43153d5f21e9980075e9d122/vendor/sigs.k8s.io/sig-storage-local-static-provisioner/pkg/deleter/deleter.go#L72 Version-Release number of selected component (if applicable): Local Storage Operator 4.8 How reproducible: Most of the time Steps to Reproduce: 1. Create a bunch of deployments that mount volumes exposed by LSO (we're creating 25 deployments) 2. Delete the deployments and the PVCs 3. Wait for PVs to be cleaned (or if you hit the "bug" they will remain in Released) Actual results: Sometimes PVs remain in "Released" state for long periods >5minutes Expected results: After PVC bound to a PV is removed, the PV should be cleaned right away. Additional info: Custom logs we added before the if[3]: syseng - pv local-pv-8fa3d5af syseng - Phase (Available) is not released (Released) As you can see in the PV cache the PV phase is "Available", but in etcd is "Released". PV status: oc get pv local-pv-8fa3d5af NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-8fa3d5af 30Gi RWO Delete Released test-lso/pvc-21 lso-filesystemclass 8m17s
I just wanted to let you know that I was able to reproduce the issue in an OCP 4.8 multi-node cluster.
Note to QA: prepare a lot of local volumes on a *single machine* (I used 4 EBS disks, each with 10 partitions, more is better). Then follow the steps to reproduce - create LocalVolume that matches them, created 40 PVCs + pods, delete all pods + pvcs. Do it few times in a loop. Some PVs will get Released and not wiped and re-created for quite some time (like minutes).
[QA Summary] [Testing Environment] - SNO deploy on top of baremetal server with the following specs: ~~~ 2 x Intel Xeon E5-2650 v4 @2.2GHz 256GB RAM 6 x 480GB SSD 2 x 10Gbps Bonded Ports ~~~ - Cluster details: ~~~ $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-10-04-203429 True False 1h5m Cluster version is 4.8.0-0.nightly-2021-10-04-203429 $ oc get infrastructures cluster -ojsonpath="{.status.infrastructureTopology}{'\n'}" SingleReplica [root@master-00 /]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 447.1G 0 disk |-sda1 8:1 0 1M 0 part |-sda2 8:2 0 127M 0 part |-sda3 8:3 0 384M 0 part /boot `-sda4 8:4 0 446.6G 0 part `-root 253:0 0 446.6G 0 crypt /sysroot sdb 8:16 0 447.1G 0 disk |-sdb1 8:17 0 2G 0 part |-sdb2 8:18 0 2G 0 part |-sdb3 8:19 0 2G 0 part `-sdb4 8:20 0 2G 0 part sdc 8:32 0 447.1G 0 disk |-sdc1 8:33 0 2G 0 part |-sdc2 8:34 0 2G 0 part |-sdc3 8:35 0 2G 0 part `-sdc4 8:36 0 2G 0 part sdd 8:48 0 447.1G 0 disk |-sdd1 8:49 0 2G 0 part |-sdd2 8:50 0 2G 0 part |-sdd3 8:51 0 2G 0 part `-sdd4 8:52 0 2G 0 part sde 8:64 0 447.1G 0 disk |-sde1 8:65 0 2G 0 part |-sde2 8:66 0 2G 0 part |-sde3 8:67 0 2G 0 part `-sde4 8:68 0 2G 0 part sdf 8:80 0 447.1G 0 disk |-sdf1 8:81 0 2G 0 part |-sdf2 8:82 0 2G 0 part |-sdf3 8:83 0 2G 0 part `-sdf4 8:84 0 2G 0 part ~~~ - Deployed Local Storage Operator with "LocalVolumeSet" for automated partition discovery and provisioning: ~~~ apiVersion: local.storage.openshift.io/v1alpha1 kind: LocalVolumeSet metadata: name: local-sc-autoset namespace: openshift-local-storage spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists storageClassName: local-sc volumeMode: Filesystem fstype: xfs deviceInclusionSpec: deviceTypes: - part deviceMechanicalProperties: - NonRotational - Rotational minSize: 1G ~~~ - Resulting SC and PVs: ~~~ $ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-sc (default) kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 175m $ oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-20f26ecb 2Gi RWO Delete Available local-sc 97s local-pv-45ef85f5 2Gi RWO Delete Available local-sc 97s local-pv-5b4aff1a 2Gi RWO Delete Available local-sc 97s local-pv-685fc0ff 2Gi RWO Delete Available local-sc 35s local-pv-783ff04a 2Gi RWO Delete Available local-sc 97s local-pv-796654fc 2Gi RWO Delete Available local-sc 66s local-pv-88c6ff28 2Gi RWO Delete Available local-sc 5s local-pv-8f430211 2Gi RWO Delete Available local-sc 97s local-pv-8fd0803 2Gi RWO Delete Available local-sc 5s local-pv-956dfb7b 2Gi RWO Delete Available local-sc 97s local-pv-a161ae47 2Gi RWO Delete Available local-sc 66s local-pv-a32b6154 2Gi RWO Delete Available local-sc 97s local-pv-a8ed57c0 2Gi RWO Delete Available local-sc 97s local-pv-abb26349 2Gi RWO Delete Available local-sc 4s local-pv-af23ea2d 2Gi RWO Delete Available local-sc 66s local-pv-c0da4610 2Gi RWO Delete Available local-sc 35s local-pv-cf151642 2Gi RWO Delete Available local-sc 66s local-pv-f12fb796 2Gi RWO Delete Available local-sc 4s local-pv-f88c49d1 2Gi RWO Delete Available local-sc 36s local-pv-feb8c0da 2Gi RWO Delete Available local-sc 36s ~~~ [Results] - Scripting deploy of 20 new projects and corresponding applications with their persistent volumes: ~~~ $ for i in {1..20}; do echo; echo "Creating project 'test$i'..."; oc new-project test$i > /dev/null; echo "Deploying 'cakephp-mysql-persistent'..."; oc new-app cakephp-mysql-persistent > /dev/null; done; echo; echo "Waiting for deploys to finish and PVCs to get bound..."; echo; sleep 120; oc get pvc -A; for i in {1..20}; do echo; echo "Deleting project 'test$i'..."; oc delete project test$i; done; sleep 3; oc get pvc -A; echo; echo "Waiting 1min to check PVCs status after deletion (there should be none)..."; echo; sleep 60; oc get pvc -A Creating project 'test1'... Deploying 'cakephp-mysql-persistent'... Creating project 'test2'... Deploying 'cakephp-mysql-persistent'... Creating project 'test3'... Deploying 'cakephp-mysql-persistent'... Creating project 'test4'... Deploying 'cakephp-mysql-persistent'... Creating project 'test5'... Deploying 'cakephp-mysql-persistent'... Creating project 'test6'... Deploying 'cakephp-mysql-persistent'... Creating project 'test7'... Deploying 'cakephp-mysql-persistent'... Creating project 'test8'... Deploying 'cakephp-mysql-persistent'... Creating project 'test9'... Deploying 'cakephp-mysql-persistent'... Creating project 'test10'... Deploying 'cakephp-mysql-persistent'... Creating project 'test11'... Deploying 'cakephp-mysql-persistent'... Creating project 'test12'... Deploying 'cakephp-mysql-persistent'... Creating project 'test13'... Deploying 'cakephp-mysql-persistent'... Creating project 'test14'... Deploying 'cakephp-mysql-persistent'... Creating project 'test15'... Deploying 'cakephp-mysql-persistent'... Creating project 'test16'... Deploying 'cakephp-mysql-persistent'... Creating project 'test17'... Deploying 'cakephp-mysql-persistent'... Creating project 'test18'... Deploying 'cakephp-mysql-persistent'... Creating project 'test19'... Deploying 'cakephp-mysql-persistent'... Creating project 'test20'... Deploying 'cakephp-mysql-persistent'... Waiting for deploys to finish and PVCs to get bound... NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test1 mysql Bound local-pv-956dfb7b 2Gi RWO local-sc 4m3s test10 mysql Bound local-pv-8f430211 2Gi RWO local-sc 3m15s test11 mysql Bound local-pv-feb8c0da 2Gi RWO local-sc 3m5s test12 mysql Bound local-pv-8fd0803 2Gi RWO local-sc 2m58s test13 mysql Bound local-pv-5b4aff1a 2Gi RWO local-sc 2m50s test14 mysql Bound local-pv-abb26349 2Gi RWO local-sc 2m44s test15 mysql Bound local-pv-685fc0ff 2Gi RWO local-sc 2m35s test16 mysql Bound local-pv-af23ea2d 2Gi RWO local-sc 2m29s test17 mysql Bound local-pv-a32b6154 2Gi RWO local-sc 2m23s test18 mysql Bound local-pv-c0da4610 2Gi RWO local-sc 2m16s test19 mysql Bound local-pv-f12fb796 2Gi RWO local-sc 2m9s test2 mysql Bound local-pv-45ef85f5 2Gi RWO local-sc 4m1s test20 mysql Bound local-pv-f88c49d1 2Gi RWO local-sc 2m1s test3 mysql Bound local-pv-20f26ecb 2Gi RWO local-sc 3m58s test4 mysql Bound local-pv-a8ed57c0 2Gi RWO local-sc 3m55s test5 mysql Bound local-pv-a161ae47 2Gi RWO local-sc 3m51s test6 mysql Bound local-pv-cf151642 2Gi RWO local-sc 3m43s test7 mysql Bound local-pv-783ff04a 2Gi RWO local-sc 3m37s test8 mysql Bound local-pv-796654fc 2Gi RWO local-sc 3m30s test9 mysql Bound local-pv-88c6ff28 2Gi RWO local-sc 3m21s Deleting project 'test1'... project.project.openshift.io "test1" deleted Deleting project 'test2'... project.project.openshift.io "test2" deleted Deleting project 'test3'... project.project.openshift.io "test3" deleted Deleting project 'test4'... project.project.openshift.io "test4" deleted Deleting project 'test5'... project.project.openshift.io "test5" deleted Deleting project 'test6'... project.project.openshift.io "test6" deleted Deleting project 'test7'... project.project.openshift.io "test7" deleted Deleting project 'test8'... project.project.openshift.io "test8" deleted Deleting project 'test9'... project.project.openshift.io "test9" deleted Deleting project 'test10'... project.project.openshift.io "test10" deleted Deleting project 'test11'... project.project.openshift.io "test11" deleted Deleting project 'test12'... project.project.openshift.io "test12" deleted Deleting project 'test13'... project.project.openshift.io "test13" deleted Deleting project 'test14'... project.project.openshift.io "test14" deleted Deleting project 'test15'... project.project.openshift.io "test15" deleted Deleting project 'test16'... project.project.openshift.io "test16" deleted Deleting project 'test17'... project.project.openshift.io "test17" deleted Deleting project 'test18'... project.project.openshift.io "test18" deleted Deleting project 'test19'... project.project.openshift.io "test19" deleted Deleting project 'test20'... project.project.openshift.io "test20" deleted NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test1 mysql Terminating local-pv-956dfb7b 2Gi RWO local-sc 4m14s test12 mysql Bound local-pv-8fd0803 2Gi RWO local-sc 3m9s test13 mysql Bound local-pv-5b4aff1a 2Gi RWO local-sc 3m1s test16 mysql Bound local-pv-af23ea2d 2Gi RWO local-sc 2m40s test17 mysql Bound local-pv-a32b6154 2Gi RWO local-sc 2m34s test18 mysql Bound local-pv-c0da4610 2Gi RWO local-sc 2m27s test19 mysql Bound local-pv-f12fb796 2Gi RWO local-sc 2m20s test2 mysql Terminating local-pv-45ef85f5 2Gi RWO local-sc 4m12s test20 mysql Bound local-pv-f88c49d1 2Gi RWO local-sc 2m12s test4 mysql Terminating local-pv-a8ed57c0 2Gi RWO local-sc 4m6s test6 mysql Terminating local-pv-cf151642 2Gi RWO local-sc 3m54s test8 mysql Terminating local-pv-796654fc 2Gi RWO local-sc 3m41s test9 mysql Terminating local-pv-88c6ff28 2Gi RWO local-sc 3m32s Waiting 1min to check PVCs status after deletion (there should be none)... No resources found ~~~ *** PASSED ***
My apologies, just noted that the target of this BZ is 4.10 and I have used 4.8.0-0.nightly for the verification, it shouldn't change the results because this PR was even tested before merging the changes, but please let me rerun the same test with latest 4.10.0-0.nightly to be on the safer side. Best Regards.
**IMPORTANT**: moving back to "ON_QA" state while still testing, sorry for the inconvenience. 4.10.0-0.nightly is using "registry.redhat.io/redhat/redhat-operator-index:v4.9" for the Operator catalog, and this index doesn't contain "local-storage-operator" yet. Jan, is ok if we test these changes using "registry.redhat.io/redhat/redhat-operator-index:v4.8" on OCP 4.10?
NVM, the proper catalog for 4.10 is "quay.io/gnufied/gnufied-index:sep30-451": ~~~ grpcurl -plaintext -d '{"name":"local-storage-operator"}' localhost:50051 api.Registry/GetPackage { "name": "local-storage-operator", "channels": [ { "name": "preview", "csvName": "local-storage-operator.v4.10.0" } ], "defaultChannelName": "preview" } ~~~ I'm proceeding with this one for 4.10 testing, I'll update with my results ASAP, thanks for your patience.
[New QA Summary] [Environment] - SNO deploy on top of baremetal server with the following specs: ~~~ 2 x Intel Xeon E5-2650 v4 @2.2GHz 256GB RAM 6 x 480GB SSD 1 x 150GB SSD 2 x 10Gbps Bonded Ports ~~~ - Cluster details: ~~~ $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2021-10-05-151518 True False 4h34m Cluster version is 4.10.0-0.nightly-2021-10-05-151518 $ oc get nodes NAME STATUS ROLES AGE VERSION master-00.pamoedom-bz2001605.qe.devcluster.openshift.com Ready master,worker 4h39m v1.22.0-rc.0+1bcce0f $ oc get infrastructures cluster -o jsonpath="{.status.infrastructureTopology}{'\n'}" SingleReplica [root@master-00 /]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 447.1G 0 disk |-sda1 8:1 0 1M 0 part |-sda2 8:2 0 127M 0 part |-sda3 8:3 0 384M 0 part /boot `-sda4 8:4 0 446.6G 0 part /sysroot sdb 8:16 0 447.1G 0 disk |-sdb1 8:17 0 2G 0 part |-sdb2 8:18 0 2G 0 part |-sdb3 8:19 0 2G 0 part `-sdb4 8:20 0 2G 0 part sdc 8:32 0 447.1G 0 disk |-sdc1 8:33 0 2G 0 part |-sdc2 8:34 0 2G 0 part |-sdc3 8:35 0 2G 0 part `-sdc4 8:36 0 2G 0 part sdd 8:48 0 447.1G 0 disk |-sdd1 8:49 0 2G 0 part |-sdd2 8:50 0 2G 0 part |-sdd3 8:51 0 2G 0 part `-sdd4 8:52 0 2G 0 part sde 8:64 0 447.1G 0 disk |-sde1 8:65 0 2G 0 part |-sde2 8:66 0 2G 0 part |-sde3 8:67 0 2G 0 part `-sde4 8:68 0 2G 0 part sdf 8:80 0 447.1G 0 disk |-sdf1 8:81 0 2G 0 part |-sdf2 8:82 0 2G 0 part |-sdf3 8:83 0 2G 0 part `-sdf4 8:84 0 2G 0 part sdg 8:96 0 139.8G 0 disk |-sdg1 8:97 0 2G 0 part |-sdg2 8:98 0 2G 0 part |-sdg3 8:99 0 2G 0 part `-sdg4 8:100 0 2G 0 part ~~~ - Deployed Local Storage Operator from catalog "quay.io/gnufied/gnufied-index:sep30-451" with "LocalVolume" for disk/partition provisioning using the following yaml files: ~~~ apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: my-operator-catalog namespace: openshift-marketplace spec: sourceType: grpc image: quay.io/gnufied/gnufied-index:sep30-451 displayName: My Operator Catalog publisher: test updateStrategy: registryPoll: interval: 30m --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: local-operator-group namespace: openshift-local-storage spec: targetNamespaces: - openshift-local-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: local-storage-operator namespace: openshift-local-storage spec: installPlanApproval: Automatic name: local-storage-operator source: my-operator-catalog sourceNamespace: openshift-marketplace --- apiVersion: "local.storage.openshift.io/v1" kind: "LocalVolume" metadata: name: "local-disks" namespace: "openshift-local-storage" spec: nodeSelector: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/worker operator: Exists storageClassDevices: - storageClassName: "local-sc" volumeMode: Filesystem fsType: xfs devicePaths: - /dev/sdb1 - /dev/sdb2 - /dev/sdb3 - /dev/sdb4 - /dev/sdc1 - /dev/sdc2 - /dev/sdc3 - /dev/sdc4 - /dev/sdd1 - /dev/sdd2 - /dev/sdd3 - /dev/sdd4 - /dev/sde1 - /dev/sde2 - /dev/sde3 - /dev/sde4 - /dev/sdf1 - /dev/sdf2 - /dev/sdf3 - /dev/sdf4 - /dev/sdg1 - /dev/sdg2 - /dev/sdg3 - /dev/sdg4 ~~~ NOTE: Instead of "LocalVolume" with manual path configuration, another option could be using "Automating discovery and provisioning for local storage devices"[1], but being still in tech preview I preferred to avoid including more variables on this occasion. [1] - https://docs.openshift.com/container-platform/4.8/storage/persistent_storage/persistent-storage-local.html#local-storage-discovery_persistent-storage-local - Results of Local SC deploy: ~~~ $ oc get csv -A NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-local-storage local-storage-operator.v4.10.0 Local Storage 4.10.0 Succeeded openshift-operator-lifecycle-manager packageserver Package Server 0.18.3 Succeeded $ oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE local-pv-13d7e106 2Gi RWO Delete Available local-sc 12s local-pv-14218c60 2Gi RWO Delete Available local-sc 12s local-pv-16204b6b 2Gi RWO Delete Available local-sc 12s local-pv-4828b971 2Gi RWO Delete Available local-sc 12s local-pv-57febc82 2Gi RWO Delete Available local-sc 12s local-pv-586c7d43 2Gi RWO Delete Available local-sc 12s local-pv-596ae921 2Gi RWO Delete Available local-sc 12s local-pv-59dbdc8f 2Gi RWO Delete Available local-sc 12s local-pv-5a8669ce 2Gi RWO Delete Available local-sc 12s local-pv-5eb9a671 2Gi RWO Delete Available local-sc 12s local-pv-6180021d 2Gi RWO Delete Available local-sc 12s local-pv-6a603763 2Gi RWO Delete Available local-sc 12s local-pv-7314ba22 2Gi RWO Delete Available local-sc 12s local-pv-7c7f7416 2Gi RWO Delete Available local-sc 12s local-pv-7e138ff4 2Gi RWO Delete Available local-sc 12s local-pv-849ce6e5 2Gi RWO Delete Available local-sc 12s local-pv-94a47cf4 2Gi RWO Delete Available local-sc 12s local-pv-ac295714 2Gi RWO Delete Available local-sc 12s local-pv-ad269b4a 2Gi RWO Delete Available local-sc 12s local-pv-b7a92547 2Gi RWO Delete Available local-sc 12s local-pv-c638fbfc 2Gi RWO Delete Available local-sc 12s local-pv-d6e900cc 2Gi RWO Delete Available local-sc 12s local-pv-df8c8b35 2Gi RWO Delete Available local-sc 12s local-pv-e9b74e57 2Gi RWO Delete Available local-sc 12s ~~~ [Results] ~~~ $ oc patch storageclass local-sc -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}' storageclass.storage.k8s.io/local-sc patched $ for i in {1..24}; do echo; echo "Creating project 'test$i'..."; oc new-project test$i > /dev/null; echo "Deploying 'cakephp-mysql-persistent'..."; oc new-app cakephp-mysql-persistent > /dev/null; done; echo; echo "Waiting for deploys to finish and PVCs to get bound..."; echo; sleep 120; oc get pvc -A; for i in {1..24}; do echo; echo "Deleting project 'test$i'..."; oc delete project test$i; done; sleep 3; oc get pvc -A; echo; echo "Waiting 1min to check PVCs status after deletion (there should be none)..."; echo; sleep 60; oc get pvc -A Creating project 'test1'... Deploying 'cakephp-mysql-persistent'... Creating project 'test2'... Deploying 'cakephp-mysql-persistent'... Creating project 'test3'... Deploying 'cakephp-mysql-persistent'... Creating project 'test4'... Deploying 'cakephp-mysql-persistent'... Creating project 'test5'... Deploying 'cakephp-mysql-persistent'... Creating project 'test6'... Deploying 'cakephp-mysql-persistent'... Creating project 'test7'... Deploying 'cakephp-mysql-persistent'... Creating project 'test8'... Deploying 'cakephp-mysql-persistent'... Creating project 'test9'... Deploying 'cakephp-mysql-persistent'... Creating project 'test10'... Deploying 'cakephp-mysql-persistent'... Creating project 'test11'... Deploying 'cakephp-mysql-persistent'... Creating project 'test12'... Deploying 'cakephp-mysql-persistent'... Creating project 'test13'... Deploying 'cakephp-mysql-persistent'... Creating project 'test14'... Deploying 'cakephp-mysql-persistent'... Creating project 'test15'... Deploying 'cakephp-mysql-persistent'... Creating project 'test16'... Deploying 'cakephp-mysql-persistent'... Creating project 'test17'... Deploying 'cakephp-mysql-persistent'... Creating project 'test18'... Deploying 'cakephp-mysql-persistent'... Creating project 'test19'... Deploying 'cakephp-mysql-persistent'... Creating project 'test20'... Deploying 'cakephp-mysql-persistent'... Creating project 'test21'... Deploying 'cakephp-mysql-persistent'... Creating project 'test22'... Deploying 'cakephp-mysql-persistent'... Creating project 'test23'... Deploying 'cakephp-mysql-persistent'... Creating project 'test24'... Deploying 'cakephp-mysql-persistent'... Waiting for deploys to finish and PVCs to get bound... NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test1 mysql Bound local-pv-7c7f7416 2Gi RWO local-sc 3m4s test10 mysql Bound local-pv-6180021d 2Gi RWO local-sc 2m41s test11 mysql Bound local-pv-94a47cf4 2Gi RWO local-sc 2m38s test12 mysql Bound local-pv-5eb9a671 2Gi RWO local-sc 2m35s test13 mysql Bound local-pv-e9b74e57 2Gi RWO local-sc 2m33s test14 mysql Bound local-pv-16204b6b 2Gi RWO local-sc 2m30s test15 mysql Bound local-pv-57febc82 2Gi RWO local-sc 2m26s test16 mysql Bound local-pv-ad269b4a 2Gi RWO local-sc 2m23s test17 mysql Bound local-pv-13d7e106 2Gi RWO local-sc 2m21s test18 mysql Bound local-pv-849ce6e5 2Gi RWO local-sc 2m18s test19 mysql Bound local-pv-59dbdc8f 2Gi RWO local-sc 2m15s test2 mysql Bound local-pv-7314ba22 2Gi RWO local-sc 3m2s test20 mysql Bound local-pv-df8c8b35 2Gi RWO local-sc 2m13s test21 mysql Bound local-pv-b7a92547 2Gi RWO local-sc 2m10s test22 mysql Bound local-pv-14218c60 2Gi RWO local-sc 2m7s test23 mysql Bound local-pv-c638fbfc 2Gi RWO local-sc 2m4s test24 mysql Bound local-pv-5a8669ce 2Gi RWO local-sc 2m1s test3 mysql Bound local-pv-7e138ff4 2Gi RWO local-sc 3m test4 mysql Bound local-pv-586c7d43 2Gi RWO local-sc 2m57s test5 mysql Bound local-pv-4828b971 2Gi RWO local-sc 2m55s test6 mysql Bound local-pv-6a603763 2Gi RWO local-sc 2m52s test7 mysql Bound local-pv-596ae921 2Gi RWO local-sc 2m49s test8 mysql Bound local-pv-ac295714 2Gi RWO local-sc 2m46s test9 mysql Bound local-pv-d6e900cc 2Gi RWO local-sc 2m43s Deleting project 'test1'... project.project.openshift.io "test1" deleted Deleting project 'test2'... project.project.openshift.io "test2" deleted Deleting project 'test3'... project.project.openshift.io "test3" deleted Deleting project 'test4'... project.project.openshift.io "test4" deleted Deleting project 'test5'... project.project.openshift.io "test5" deleted Deleting project 'test6'... project.project.openshift.io "test6" deleted Deleting project 'test7'... project.project.openshift.io "test7" deleted Deleting project 'test8'... project.project.openshift.io "test8" deleted Deleting project 'test9'... project.project.openshift.io "test9" deleted Deleting project 'test10'... project.project.openshift.io "test10" deleted Deleting project 'test11'... project.project.openshift.io "test11" deleted Deleting project 'test12'... project.project.openshift.io "test12" deleted Deleting project 'test13'... project.project.openshift.io "test13" deleted Deleting project 'test14'... project.project.openshift.io "test14" deleted Deleting project 'test15'... project.project.openshift.io "test15" deleted Deleting project 'test16'... project.project.openshift.io "test16" deleted Deleting project 'test17'... project.project.openshift.io "test17" deleted Deleting project 'test18'... project.project.openshift.io "test18" deleted Deleting project 'test19'... project.project.openshift.io "test19" deleted Deleting project 'test20'... project.project.openshift.io "test20" deleted Deleting project 'test21'... project.project.openshift.io "test21" deleted Deleting project 'test22'... project.project.openshift.io "test22" deleted Deleting project 'test23'... project.project.openshift.io "test23" deleted Deleting project 'test24'... project.project.openshift.io "test24" deleted NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test1 mysql Terminating local-pv-7c7f7416 2Gi RWO local-sc 3m18s test10 mysql Terminating local-pv-6180021d 2Gi RWO local-sc 2m55s test11 mysql Bound local-pv-94a47cf4 2Gi RWO local-sc 2m52s test12 mysql Bound local-pv-5eb9a671 2Gi RWO local-sc 2m49s test13 mysql Bound local-pv-e9b74e57 2Gi RWO local-sc 2m47s test14 mysql Bound local-pv-16204b6b 2Gi RWO local-sc 2m44s test15 mysql Bound local-pv-57febc82 2Gi RWO local-sc 2m40s test16 mysql Bound local-pv-ad269b4a 2Gi RWO local-sc 2m37s test17 mysql Bound local-pv-13d7e106 2Gi RWO local-sc 2m35s test18 mysql Bound local-pv-849ce6e5 2Gi RWO local-sc 2m32s test19 mysql Bound local-pv-59dbdc8f 2Gi RWO local-sc 2m29s test2 mysql Terminating local-pv-7314ba22 2Gi RWO local-sc 3m16s test20 mysql Bound local-pv-df8c8b35 2Gi RWO local-sc 2m27s test21 mysql Bound local-pv-b7a92547 2Gi RWO local-sc 2m24s test22 mysql Bound local-pv-14218c60 2Gi RWO local-sc 2m21s test23 mysql Bound local-pv-c638fbfc 2Gi RWO local-sc 2m18s test24 mysql Bound local-pv-5a8669ce 2Gi RWO local-sc 2m15s test3 mysql Terminating local-pv-7e138ff4 2Gi RWO local-sc 3m14s test4 mysql Terminating local-pv-586c7d43 2Gi RWO local-sc 3m11s test5 mysql Terminating local-pv-4828b971 2Gi RWO local-sc 3m9s test6 mysql Terminating local-pv-6a603763 2Gi RWO local-sc 3m6s test7 mysql Terminating local-pv-596ae921 2Gi RWO local-sc 3m3s test8 mysql Terminating local-pv-ac295714 2Gi RWO local-sc 3m test9 mysql Terminating local-pv-d6e900cc 2Gi RWO local-sc 2m57s Waiting 1min to check PVCs status after deletion (there should be none)... No resources found (repeat) ~~~ NOTE: The for loop was executed 5 times on a row and no problem was detected. *** PASSED ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056