Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1737389

Summary: CSI: MountOption in storage class does not take effect for ebs volume
Product: OpenShift Container Platform Reporter: Chao Yang <chaoyang>
Component: StorageAssignee: Fabio Bertinatto <fbertina>
Status: CLOSED ERRATA QA Contact: Chao Yang <chaoyang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aos-bugs, aos-storage-staff, bchilds, nagrawal
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:34:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chao Yang 2019-08-05 09:40:06 UTC 
Description of problem:
MountOption in storage class does not take effect for ebs volume

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-07-31-162901

How reproducible:
Always

Steps to Reproduce:
1.Create sc
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: sc-mount
parameters:
  type: gp2
  fsType: xfs
  csi.storage.k8s.io/provisioner-secret-name: aws-creds
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - "discard"
2.Create pvc and pod
3.Waiting for pod is running
4.oc rsh mypod-mount

/dev/nvme3n1 on /tmp type xfs (rw,seclabel,relatime,attr2,inode64,noquota)


Actual results:
No discard showed when check mount option

Expected results:
"discard" should be added when mount volumes

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Comment 1 Jan Safranek 2019-08-06 09:05:08 UTC 
I checked that external-provisioner passes the mount options to provisioned PVs:

- apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/provisioned-by: ebs.csi.aws.com
    name: pvc-ee2b2484-b827-11e9-8a50-068fac16b29a
  spec:
    csi:
      driver: ebs.csi.aws.com
      fsType: ext4
      volumeAttributes:
        fstype: ""
        storage.kubernetes.io/csiProvisionerIdentity: 1565017758875-8081-ebs.csi.aws.com
      volumeHandle: vol-07272e50cd39021ab
    mountOptions:
    - discard
  ...


I noticed that NodeStageVolume is called without mount options:

I0806 08:56:09.189379       1 node.go:93] NodeStageVolume: called with args {VolumeId:vol-07272e50cd39021ab PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ee2b2484-b827-11e9-8a50-068fac16b29a/globalmount VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[] VolumeContext:map[fstype: storage.kubernetes.io/csiProvisionerIdentity:1565017758875-8081-ebs.csi.aws.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}


NodePublish is called with mount options, but it's too late:
I0806 08:56:09.465438       1 node.go:267] NodePublishVolume: called with args {VolumeId:vol-07272e50cd39021ab PublishContext:map[devicePath:/dev/xvdba] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ee2b2484-b827-11e9-8a50-068fac16b29a/globalmount TargetPath:/var/lib/kubelet/pods/03c8c3a9-b828-11e9-bca8-02c1d4176b00/volumes/kubernetes.io~csi/pvc-ee2b2484-b827-11e9-8a50-068fac16b29a/mount VolumeCapability:mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[fstype: storage.kubernetes.io/csiProvisionerIdentity:1565017758875-8081-ebs.csi.aws.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
Comment 2 Jan Safranek 2019-08-06 09:17:18 UTC 
This seems to be fixed by https://github.com/kubernetes/kubernetes/pull/80191
Comment 4 Chao Yang 2019-08-20 05:51:00 UTC 
It is failed on 4.2.0-0.nightly-2019-08-19-201622

Checking mount options in pod:
/dev/xvdbb on /tmp type ext4 (rw,seclabel,relatime)


I0820 05:43:51.923559       1 node.go:93] NodeStageVolume: called with args {VolumeId:vol-0bdb9d6a3bc3ce021 PublishContext:map[devicePath:/dev/xvdbb] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount VolumeCapability:mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1566270480235-8081-ebs.csi.aws.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0820 05:43:51.923691       1 node.go:148] NodeStageVolume: find device path /dev/xvdbb -> /dev/xvdbb
I0820 05:43:51.924271       1 node.go:183] NodeStageVolume: formatting /dev/xvdbb and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount with fstype ext4
I0820 05:43:51.924298       1 mount_linux.go:441] Checking for issues with fsck on disk: /dev/xvdbb
I0820 05:43:51.943455       1 mount_linux.go:454] `fsck` error fsck from util-linux 2.30.2
fsck.ext2: Bad magic number in super-block while trying to open /dev/xvdbb
/dev/xvdbb: 
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I0820 05:43:51.943479       1 mount_linux.go:460] Attempting to mount disk: ext4 /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount
I0820 05:43:51.943498       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount])
E0820 05:43:51.950021       1 mount_linux.go:147] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount: wrong fs type, bad option, bad superblock on /dev/xvdbb, missing codepage or helper program, or other error.

I0820 05:43:51.950050       1 mount_linux.go:515] Attempting to determine if disk "/dev/xvdbb" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/xvdbb])
I0820 05:43:51.960530       1 mount_linux.go:518] Output: "", err: exit status 2
I0820 05:43:51.960557       1 mount_linux.go:489] Disk "/dev/xvdbb" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/xvdbb]
I0820 05:43:52.078850       1 mount_linux.go:493] Disk successfully formatted (mkfs): ext4 - /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount
I0820 05:43:52.078893       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount])
I0820 05:43:52.088517       1 node.go:134] NodeStageVolume: volume="vol-0bdb9d6a3bc3ce021" operation finished
I0820 05:43:52.089999       1 node.go:339] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0820 05:43:52.097090       1 node.go:269] NodePublishVolume: called with args {VolumeId:vol-0bdb9d6a3bc3ce021 PublishContext:map[devicePath:/dev/xvdbb] StagingTargetPath:/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount TargetPath:/var/lib/kubelet/pods/74b0572e-c30d-11e9-9335-0254d8b891de/volumes/kubernetes.io~csi/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/mount VolumeCapability:mount:<fs_type:"ext4" mount_flags:"discard" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1566270480235-8081-ebs.csi.aws.com] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0820 05:43:52.097169       1 node.go:439] NodePublishVolume: creating dir /var/lib/kubelet/pods/74b0572e-c30d-11e9-9335-0254d8b891de/volumes/kubernetes.io~csi/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/mount
I0820 05:43:52.097197       1 node.go:449] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount at /var/lib/kubelet/pods/74b0572e-c30d-11e9-9335-0254d8b891de/volumes/kubernetes.io~csi/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/mount with option [bind discard] as fstype ext4
I0820 05:43:52.097221       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-t ext4 -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount /var/lib/kubelet/pods/74b0572e-c30d-11e9-9335-0254d8b891de/volumes/kubernetes.io~csi/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/mount])
I0820 05:43:52.099048       1 mount_linux.go:142] Mounting cmd (mount) with arguments ([-t ext4 -o bind,remount,discard /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/globalmount /var/lib/kubelet/pods/74b0572e-c30d-11e9-9335-0254d8b891de/volumes/kubernetes.io~csi/pvc-694edcbb-c30d-11e9-9260-0af72f45a844/mount])
Comment 12 errata-xmlrpc 2019-10-16 06:34:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922