Bug 1948610
| Summary: | External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly] | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qin Ping <piqin> |
| Component: | Storage | Assignee: | Fabio Bertinatto <fbertina> |
| Storage sub component: | Operators | QA Contact: | Chao Yang <chaoyang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs, chaoyang, fbertina |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-27 22:59:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I believe this has been fixed by this PR [1]. Looking at the latest jobs, it seems like this test is no longer failing. [1] https://github.com/openshift/azure-disk-csi-driver-operator/pull/13 Check the recently e2e ci jobs, didn't find the error. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: fail [k8s.io/kubernetes.0/test/e2e/storage/testsuites/multivolume.go:415]: Unexpected error: <*errors.errorString | 0xc001533940>: { s: "pod \"pod-64c36f12-253b-44bd-ad99-ff0f7219a195\" is not Running: timed out waiting for the condition", } pod "pod-64c36f12-253b-44bd-ad99-ff0f7219a195" is not Running: timed out waiting for the condition occurred failed: (5m59s) 2021-04-12T15:04:10 "External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly]" Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-04-09-222447 How reproducible: about 50% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: Events of Pods: 28s Warning FailedMapVolume pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195 MapVolume.MapPodDevice failed for volume "pvc-0c6668b8-b074-414a-936a-64b719e915b7" : rpc error: code = DeadlineExceeded desc = context deadline exceeded 2m29s Warning FailedMapVolume pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195 MapVolume.SetUpDevice failed for volume "pvc-0c6668b8-b074-414a-936a-64b719e915b7" : rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists 74s Warning FailedMount pod/pod-64c36f12-253b-44bd-ad99-ff0f7219a195 Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-hrvrx volume1 volume2]: timed out waiting for the condition logs of csi driver node pod: $ oc -n openshift-cluster-csi-drivers logs azure-disk-csi-driver-node-529wq -c csi-driver|grep pvc-0c6668b8-b074-414a-936a-64b719e915b7 I0412 14:59:28.734305 1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 14:59:28.739505 1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:01:29.340589 1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} E0412 15:01:29.340744 1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists I0412 15:01:30.361669 1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:01:30.367298 1 utils.go:95] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0c6668b8-b074-414a-936a-64b719e915b7","csi.storage.k8s.io/pvc/name":"disk.csi.azure.comwlzc4","csi.storage.k8s.io/pvc/namespace":"e2e-multivolume-2214","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1618212010832-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:03:30.404680 1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:03:30.404756 1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce W0412 15:03:30.404800 1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce E0412 15:03:30.404815 1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists I0412 15:03:30.918981 1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:03:30.919058 1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce W0412 15:03:30.919090 1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce E0412 15:03:30.919101 1 utils.go:99] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 already exists I0412 15:03:31.928121 1 utils.go:95] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:03:31.928184 1 nodeserver.go:269] NodeUnpublishVolume: unmounting volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce W0412 15:03:31.928224 1 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce I0412 15:03:31.928238 1 nodeserver.go:280] NodeUnpublishVolume: unmount volume /subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-0c6668b8-b074-414a-936a-64b719e915b7/ad6b421b-7862-4565-bbce-6479680735ce successfully I0412 15:03:32.031404 1 utils.go:95] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7","volume_id":"/subscriptions/53b8f551-f0fc-4bea-8cba-6d1fefd54c8a/resourceGroups/piqin-0412-z8dh9-rg/providers/Microsoft.Compute/disks/pvc-0c6668b8-b074-414a-936a-64b719e915b7"} I0412 15:03:32.031484 1 nodeserver.go:175] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7 W0412 15:03:32.031561 1 mount_helper_common.go:65] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7" is not a mountpoint, deleting I0412 15:03:32.032065 1 nodeserver.go:180] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-0c6668b8-b074-414a-936a-64b719e915b7 successfully