Description of problem: Moutning XFS restored snapshot volume to same node is failed Upstream issue here: https://github.com/container-storage-interface/spec/issues/482 Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-05-25-223219 How reproducible: Always Steps to Reproduce: 1.Create pvc/pod using below storageclass: oc get sc foo -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "false" creationTimestamp: "2021-05-27T01:55:45Z" name: foo resourceVersion: "536907" uid: 8d77991b-6b96-4995-b044-8b232168712e parameters: fsType: xfs provisioner: ebs.csi.aws.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer 2.Create volumesnapshot oc get volumesnapshot NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE new-snapshot-test-1 true pvc1 1Gi gcp-snap-2 snapcontent-55a1ee74-6791-4506-aae7-50a093b8cb20 45m 45m 3.Create restored pvc/pod. Make sure pod is scheduled to same node. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13s default-scheduler Successfully assigned test/pod2 to ip-10-0-169-208.us-east-2.compute.internal Normal SuccessfulAttachVolume 11s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-b9076058-77c4-4496-a9be-22b7bfcddae7" Warning FailedMount 2s (x4 over 5s) kubelet MountVolume.MountDevice failed for volume "pvc-b9076058-77c4-4496-a9be-22b7bfcddae7" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount": mount failed: exit status 32 Mounting command: mount Mounting arguments: -t xfs -o defaults /dev/nvme3n1 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount: wrong fs type, bad option, bad superblock on /dev/nvme3n1, missing codepage or helper program, or other error. Actual results: Restored pod is failed due to xfs volume could not mount. Expected results: Restored pod should be running. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
This affects also cloned volumes. Since the root cause (and the fix) are the same, let's track them in a single bug.
*** Bug 1963885 has been marked as a duplicate of this bug. ***
Since the bug needs to be fixed in each affected CSI driver, I filed separate bugs for them (see Blocks field). I am closing this BZ, let me know if I missed any driver.