Bug 1752395

Summary: Could not mount azure file on ipi installed ocp
Product: OpenShift Container Platform Reporter: Chao Yang <chaoyang>
Component: RHCOSAssignee: Micah Abbott <miabbott>
Status: CLOSED ERRATA QA Contact: Wei Duan <wduan>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: aos-bugs, aos-storage-staff, apitt, bbennett, bbreard, chuffman, dcain, dornelas, dustymabe, eparis, fbertina, imcleod, jligon, llopezmo, miabbott, mnguyen, nstielau
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:41:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913    

Description Chao Yang 2019-09-16 09:29:26 UTC
Description of problem:
Could not mount azure file on ipi installed ocp

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-09-11-202233

How reproducible:
Always

Steps to Reproduce:
1.Create role and role-binding
2.Create storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-09-16T08:24:15Z"
  name: azfsc
  resourceVersion: "41012"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/azfsc
  uid: 5eac6447-d85b-11e9-be4c-000d3a975032
parameters:
  storageAccount: azurechao9x5r5g422c
provisioner: kubernetes.io/azure-file
reclaimPolicy: Delete
volumeBindingMode: Immediate

3.Create secret

4.Create pvc

5.Check pv is created
oc get pv pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: azure-file-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-file
  creationTimestamp: "2019-09-16T08:44:31Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4
  resourceVersion: "45627"
  selfLink: /api/v1/persistentvolumes/pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4
  uid: 33727459-d85e-11e9-832e-000d3a975743
spec:
  accessModes:
  - ReadWriteMany
  azureFile:
    secretName: azure-storage-account-azurechao9x5r5g422c-secret
    secretNamespace: wehe
    shareName: azure-chao-9x5r5-dynam-pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: azpvc
    namespace: wehe
    resourceVersion: "45412"
    uid: 10a063b4-d85e-11e9-bb6b-000d3a9751a4
  persistentVolumeReclaimPolicy: Delete
  storageClassName: azfsc
  volumeMode: Filesystem
status:
 phase: Bound

6.Create pod
Pod is not running.

7.oc describe pods pod3
Name:               pod3
Namespace:          wehe
Priority:           0
PriorityClassName:  <none>
Node:               azure-chao-9x5r5-worker-centralus2-sbmz6/10.0.32.4
Start Time:         Mon, 16 Sep 2019 16:53:18 +0800
Labels:             name=frontendhttp
Annotations:        openshift.io/scc: privileged
                    ovn: {"ip_address":"10.131.0.21/23", "mac_address":"2e:33:4f:83:00:16", "gateway_ip": "10.131.0.1"}
Status:             Pending
IP:                 
Containers:
  myfrontend:
    Container ID:   
    Image:          aosqe/hello-openshift
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp from aws (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-m8k78 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  aws:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  azpvc
    ReadOnly:   false
  default-token-m8k78:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-m8k78
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age   From                                               Message
  ----     ------       ----  ----                                               -------
  Normal   Scheduled    11s   default-scheduler                                  Successfully assigned wehe/pod3 to azure-chao-9x5r5-worker-centralus2-sbmz6
  Warning  FailedMount  11s   kubelet, azure-chao-9x5r5-worker-centralus2-sbmz6  MountVolume.SetUp failed for volume "pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/6d491ea3-d85f-11e9-bb6b-000d3a9751a4/volumes/kubernetes.io~azure-file/pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4 --scope -- mount -t cifs -o username=azurechao9x5r5g422c,password=7bttpOR8JeosVxB6+W4GcWI+StFu7exKaaSxclUo7utNHwkAO/hkfUuRK5XJACbmi3WY6f71lGQRe5K2Ik759g==,file_mode=0777,dir_mode=0777,vers=3.0 //azurechao9x5r5g422c.file.core.windows.net/azure-chao-9x5r5-dynam-pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4 /var/lib/kubelet/pods/6d491ea3-d85f-11e9-bb6b-000d3a9751a4/volumes/kubernetes.io~azure-file/pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4
Output: Running scope as unit: run-r8c35e7528db64ff2be4e2d5e24662dae.scope
mount: /var/lib/kubelet/pods/6d491ea3-d85f-11e9-bb6b-000d3a9751a4/volumes/kubernetes.io~azure-file/pvc-10a063b4-d85e-11e9-bb6b-000d3a9751a4: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.


Actual results:
mount.cifs: command not found

Expected results:
Pod should be running

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Fabio Bertinatto 2019-09-16 13:02:22 UTC
The binary mount.cifs is missing in RHCOS. Changing to RHCOS component.

Comment 9 errata-xmlrpc 2019-10-16 06:41:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922