Bug 1988375

+++ This bug was initially created as a clone of Bug #1965155 +++

Description of problem:
Moutning XFS restored snapshot volume to same node is failed
Upstream issue here: https://github.com/container-storage-interface/spec/issues/482

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-05-25-223219 

How reproducible:
Always

Steps to Reproduce:
1.Create pvc/pod using below storageclass:
oc get sc foo -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  creationTimestamp: "2021-05-27T01:55:45Z"
  name: foo
  resourceVersion: "536907"
  uid: 8d77991b-6b96-4995-b044-8b232168712e
parameters:
  fsType: xfs
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

2.Create volumesnapshot
oc get volumesnapshot
NAME                  READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
new-snapshot-test-1   true         pvc1                                1Gi           gcp-snap-2      snapcontent-55a1ee74-6791-4506-aae7-50a093b8cb20   45m            45m

3.Create restored pvc/pod. Make sure pod is scheduled to same node.
Events:
  Type     Reason                  Age              From                     Message
  ----     ------                  ----             ----                     -------
  Normal   Scheduled               13s              default-scheduler        Successfully assigned test/pod2 to ip-10-0-169-208.us-east-2.compute.internal
  Normal   SuccessfulAttachVolume  11s              attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b9076058-77c4-4496-a9be-22b7bfcddae7"
  Warning  FailedMount             2s (x4 over 5s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-b9076058-77c4-4496-a9be-22b7bfcddae7" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o defaults /dev/nvme3n1 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b9076058-77c4-4496-a9be-22b7bfcddae7/globalmount: wrong fs type, bad option, bad superblock on /dev/nvme3n1, missing codepage or helper program, or other error.


Actual results:
Restored pod is failed due to xfs volume could not mount.

Expected results:
Restored pod should be running.
Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

--- Additional comment from Jan Safranek on 2021-06-03 14:13:15 UTC ---

This affects also cloned volumes. Since the root cause (and the fix) are the same, let's track them in a single bug.

--- Additional comment from Jan Safranek on 2021-06-03 14:14:11 UTC ---