Bug 1969426 - E2E tests leave mock PVs behind
Summary: E2E tests leave mock PVs behind
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Tomas Smetana
QA Contact: Rohit Patil
URL:
Whiteboard:
Depends On: 1959445
Blocks: 1952931 2002665
TreeView+ depends on / blocked
 
Reported: 2021-06-08 12:39 UTC by Tomas Smetana
Modified: 2021-10-06 15:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1959445
: 2002665 (view as bug list)
Environment:
Last Closed: 2021-09-09 13:05:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 102690 0 None closed CSI e2e: stop leaking pvs in CSI mock snapshot test 2021-06-09 15:53:22 UTC

Description Tomas Smetana 2021-06-08 12:39:03 UTC
+++ This bug was initially created as a clone of Bug #1959445 +++

Description of problem:

After CSI migration tests finish, there are some leftover PVs in the cluster.

See https://testgrid.k8s.io/redhat-openshift-ocp-release-4.8-broken#periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration

From https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391694972192821248:

+persistentvolume/local-pv9zb5m
+persistentvolume/local-pvpt5mt
 persistentvolume/pvc-e3b4f183-4c75-4d9c-9385-18b088abcd1b
 persistentvolume/pvc-f15067db-d6d5-4c94-9079-d28bc37ed925
+persistentvolume/pvc-f9c9f7d0-51b8-402a-80f8-4bbc1982a6d0
ERROR: Timed out waiting for PVs to get deleted.
ERROR: It seems that some test left some PVs behind.
ERROR: Check the diff between expected and existing PVs above.

So we have three PVs left behind and not deleted. Two of them are local, the third one looks like mock:

    name: pvc-f9c9f7d0-51b8-402a-80f8-4bbc1982a6d0
...
    csi:
      driver: csi-mock-e2e-csi-mock-volumes-7168
...
    phase: Released

(from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391694972192821248/artifacts/e2e-aws-csi-migration/storage-pv-check/artifacts/pvs-29.yaml)

How reproducible:
Always

Steps to Reproduce:
Run openshift conformance/parallel tests against OCP on AWS.

CSI migration is enabled in the linked CI jobs, IMO it should not affect local or mock volumes, but one never knows. We do not check for leftover PVs in other e2e jobs, so we don't know if there are some on GCE or without migration.

Actual results:
At least two local PVs left behind (this is 100% reproducible)

Sometimes, a CSI mock PV left behind (started appearing in https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.8-e2e-aws-csi-migration/1391332579759624192 ???)

Expected results:
No PVs left behind

Comment 1 Tomas Smetana 2021-06-08 12:40:27 UTC
The local volumes will be fixed with bug #1959445

Comment 2 Tomas Smetana 2021-09-09 13:05:08 UTC
This is fixed in 4.9 with rebase to k8s 1.22:

https://github.com/openshift/kubernetes/blob/release-4.9/test/e2e/storage/csi_mock_volume.go#L1711


Note You need to log in before you can comment on or make changes to this bug.