Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2002665

Summary: E2E tests leave mock PVs behind
Product: OpenShift Container Platform Reporter: Tomas Smetana <tsmetana>
Component: StorageAssignee: Tomas Smetana <tsmetana>
Storage sub component: Kubernetes QA Contact: Rohit Patil <ropatil>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: medium CC: jsafrane, piqin, ropatil, wduan
Version: 4.8   
Target Milestone: ---   
Target Release: 4.8.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1969426 Environment:
Last Closed: 2022-08-19 10:05:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1959445, 1969426    
Bug Blocks: 1952931    

Description Tomas Smetana 2021-09-09 13:07:38 UTC 
+++ This bug was initially created as a clone of Bug #1969426 +++

+++ This bug was initially created as a clone of Bug #1959445 +++

Description of problem:

After CSI migration tests finish, there are some leftover PVs in the cluster.

See https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-broken#periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration

From https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391694972192821248:

+persistentvolume/local-pv9zb5m
+persistentvolume/local-pvpt5mt
 persistentvolume/pvc-e3b4f183-4c75-4d9c-9385-18b088abcd1b
 persistentvolume/pvc-f15067db-d6d5-4c94-9079-d28bc37ed925
+persistentvolume/pvc-f9c9f7d0-51b8-402a-80f8-4bbc1982a6d0
ERROR: Timed out waiting for PVs to get deleted.
ERROR: It seems that some test left some PVs behind.
ERROR: Check the diff between expected and existing PVs above.

So we have three PVs left behind and not deleted. Two of them are local, the third one looks like mock:

    name: pvc-f9c9f7d0-51b8-402a-80f8-4bbc1982a6d0
...
    csi:
      driver: csi-mock-e2e-csi-mock-volumes-7168
...
    phase: Released

(from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391694972192821248/artifacts/e2e-aws-csi-migration/storage-pv-check/artifacts/pvs-29.yaml)

How reproducible:
Always

Steps to Reproduce:
Run openshift conformance/parallel tests against OCP on AWS.

CSI migration is enabled in the linked CI jobs, IMO it should not affect local or mock volumes, but one never knows. We do not check for leftover PVs in other e2e jobs, so we don't know if there are some on GCE or without migration.

Actual results:
At least two local PVs left behind (this is 100% reproducible)

Sometimes, a CSI mock PV left behind (started appearing in https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391332579759624192 ???)

Expected results:
No PVs left behind

--- Additional comment from Tomas Smetana on 2021-06-08 14:40:27 CEST ---

The local volumes will be fixed with bug #1959445

--- Additional comment from Tomas Smetana on 2021-09-09 15:05:08 CEST ---

This is fixed in 4.9 with rebase to k8s 1.22:

https://github.com/openshift/kubernetes/blob/release-4.9/test/e2e/storage/csi_mock_volume.go#L1711
Comment 4 Tomas Smetana 2022-08-19 10:05:26 UTC 
Closing the bug: I'm not sure this is worth backporting. If whoever believes this should be really fixed in 4.8.z, please file a new bug in Jira: https://issues.redhat.com/secure/CreateIssueDetails!init.jspa?pid=12332330&issuetype=1