1948610 – External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly]

Bug 1948610 - External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly]

Summary: External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	4.8
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Fabio Bertinatto
QA Contact:	Chao Yang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-04-12 15:13 UTC by Qin Ping
Modified:	2021-07-27 22:59 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-07-27 22:59:27 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 22:59:43 UTC

Description Qin Ping 2021-04-12 15:13:58 UTC

Description of problem:
fail [k8s.io/kubernetes.0/test/e2e/storage/testsuites/multivolume.go:415]: Unexpected error:
    <*errors.errorString | 0xc001533940>: {
        s: "pod \"pod-64c36f12-253b-44bd-ad99-ff0f7219a195\" is not Running: timed out waiting for the condition",
    }
    pod "pod-64c36f12-253b-44bd-ad99-ff0f7219a195" is not Running: timed out waiting for the condition
occurred

failed: (5m59s) 2021-04-12T15:04:10 "External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly]"


Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-09-222447

How reproducible:
about 50%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Events of Pods:
28s         Warning   FailedMapVolume          pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195    MapVolume.MapPodDevice failed for volume "pvc-0c6668b8-b074-414a-936a-64b719e915b7" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
2m29s       Warning   FailedMapVolume          pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195    MapVolume.SetUpDevice failed for volume "pvc-0c6668b8-b074-414a-936a-64b719e915b7" : rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists
74s         Warning   FailedMount              pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195    Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-hrvrx volume1 volume2]: timed out waiting for the condition

logs of csi driver node pod:
$ oc -n openshift-cluster-csi-drivers logs azure-disk-csi-driver-node-529wq -c csi-driver|grep pvc-0c6668b8-b074-414a-936a-64b719e915b7
I0412 14:59:28.734305       1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 14:59:28.739505       1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:01:29.340589       1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
E0412 15:01:29.340744       1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists
I0412 15:01:30.361669       1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:01:30.367298       1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:03:30.404680       1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:03:30.404756       1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
W0412 15:03:30.404800       1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
E0412 15:03:30.404815       1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists
I0412 15:03:30.918981       1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:03:30.919058       1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
W0412 15:03:30.919090       1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
E0412 15:03:30.919101       1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists
I0412 15:03:31.928121       1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:03:31.928184       1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
W0412 15:03:31.928224       1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce
I0412 15:03:31.928238       1 nodeserver.go:280] NodeUnpublishVolume: unmount volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce successfully
I0412 15:03:32.031404       1 utils.go:95] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"}
I0412 15:03:32.031484       1 nodeserver.go:175] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7
W0412 15:03:32.031561       1 mount_helper_common.go:65] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7" is not a mountpoint, deleting
I0412 15:03:32.032065       1 nodeserver.go:180] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7 successfully

Comment 1 Fabio Bertinatto 2021-04-23 13:13:51 UTC

I believe this has been fixed by this PR [1]. Looking at the latest jobs, it seems like this test is no longer failing.

[1] https://github.com/openshift/azure-disk-csi-driver-operator/pull/13

Comment 3 Qin Ping 2021-04-29 01:50:59 UTC

Check the recently e2e ci jobs, didn't find the error.

Comment 6 errata-xmlrpc 2021-07-27 22:59:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.