Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1952931

Summary:	CI does not check leftover PVs
Product:	OpenShift Container Platform	Reporter:	Jan Safranek <jsafrane>
Component:	Storage	Assignee:	Jan Safranek <jsafrane>
Storage sub component:	Storage	QA Contact:	Qin Ping <piqin>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, fbertina, jsafrane, piqin
Version:	4.8
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	1948474	Environment:
Last Closed:	2021-07-27 23:03:30 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1948474, 1959445, 1969426, 2002660, 2002665
Bug Blocks:

Description Jan Safranek 2021-04-23 15:29:41 UTC

Cloning to add a CI job to check if there is a stray PV left behind the e2e tests. This should not happen and should be reported as a test failure.

+++ This bug was initially created as a clone of Bug #1948474 +++

Description of problem:
After running of csi verification test, one of PV is stuck in “Released” status

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-09-222447

How reproducible:
Hit one time, not try to reproduce it yet.

Steps to Reproduce:
1. Set Up an OCP4.8 cluster on Azure
2. Enable TechPreviewNoUpgrade featureset
3. Run csi verification test manually(with test image included in the payload image)
   # openshift-tests run openshift/csi
4. When the test is finished, check if there are some resources not cleaned

Actual results:
One of PV is stuck in “Release” status.
$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                         STORAGECLASS                      REASON   AGE
pvc-537e7872-6402-4923-b7d3-619873bf7d08   1Gi        RWO            Delete           Bound      e2e-multivolume-294/disk.csi.azure.comjcjj6   e2e-multivolume-294-e2e-scfw6h2            82m
pvc-5eb4ce13-34e2-493c-b348-91acd616dc17   1Gi        RWO            Delete           Released   e2e-volumemode-4624/disk.csi.azure.com7bkj5   e2e-volumemode-4624-e2e-sc88cnw            64m
pvc-8be0451d-e0d4-4823-a918-0e578aeb977d   1Gi        RWO            Delete           Bound      e2e-multivolume-294/disk.csi.azure.comtsmq8   e2e-multivolume-294-e2e-scf544k            82m

The PVC and the namespace includes the PVC were cleaned.
Events:
  Type     Reason              Age                  From                                                                               Message
  ----     ------              ----                 ----                                                                               -------
  Warning  VolumeFailedDelete  67m (x6 over 67m)    disk.csi.azure.com_piqin-0412-z8dh9-master-0_0909beaf-168b-4bc2-ad17-e13dd0213ed7  persistentvolume pvc-5eb4ce13-34e2-493c-b348-91acd616dc17 is still attached to node piqin-0412-z8dh9-worker-northcentralus-8tj9w
  Warning  VolumeFailedDelete  4m8s (x15 over 66m)  disk.csi.azure.com_piqin-0412-z8dh9-master-0_0909beaf-168b-4bc2-ad17-e13dd0213ed7  rpc error: code = Unknown desc = disk(/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-5eb4ce13-34e2-493c-b348-91acd616dc17) already attached to node(/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/virtualMachines/piqin-0412-z8dh9-worker-northcentralus-8tj9w), could not be deleted



Expected results:
PV can be deleted after the test.

Additional info:
csi verification test result:
Storage Capabilities (guaranteed only on full CSI test suite with 0 fails)
==========================================================================
Driver short name:                         azuredisk
Driver name:                               disk.csi.azure.com
Storage class:                             
Supported OpenShift / CSI features:
  Persistent volumes:                      true
  Raw block mode:                          true
  FSGroup:                                 true
  Executable files on a volume:            true
  Volume snapshots:                        true
  Volume cloning:                          false
  Use volume from multiple pods on a node: true
  ReadWriteMany access mode:               false
  Volume expansion for controller:         true
  Volume expansion for node:               true
  Volume limits:                           false
  Volume can run on single node:           false
  Topology:                                true
Supported OpenShift Virtualization features:
  Raw block VM disks:                      true
  Live migration:                          false
  VM snapshots:                            true
  Storage-assisted cloning:                true

error: 20 fail, 39 pass, 128 skip (36m39s)
---

As the clo

Comment 2 Qin Ping 2021-04-29 08:42:40 UTC

Verified with: 4.8.0-0.nightly-2021-04-26-151924

Storage Capabilities (guaranteed only on full CSI test suite with 0 fails)
==========================================================================
Driver short name:                         azuredisk
Driver name:                               disk.csi.azure.com
Storage class:                             
Supported OpenShift / CSI features:
  Persistent volumes:                      true
  Raw block mode:                          true
  FSGroup:                                 true
  Executable files on a volume:            true
  Volume snapshots:                        true
  Volume cloning:                          false
  Use volume from multiple pods on a node: true
  ReadWriteMany access mode:               false
  Volume expansion for controller:         true
  Volume expansion for node:               true
  Volume limits:                           false
  Volume can run on single node:           false
  Topology:                                true
Supported OpenShift Virtualization features:
  Raw block VM disks:                      true
  Live migration:                          false
  VM snapshots:                            true
  Storage-assisted cloning:                true

error: 7 fail, 51 pass, 129 skip (24m9s)
sh-4.4# 
sh-4.4# oc get storageclass
NAME                        PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
managed-csi                 disk.csi.azure.com         Delete          WaitForFirstConsumer   true                   48m
managed-premium (default)   kubernetes.io/azure-disk   Delete          WaitForFirstConsumer   true                   88m
sh-4.4# oc get ns|grep e2e
sh-4.4# oc get pv
No resources found
sh-4.4# oc get pvc -A
No resources found

Comment 5 errata-xmlrpc 2021-07-27 23:03:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438