Bug 1833437

Related bug
https://bugzilla.redhat.com/show_bug.cgi?id=1833697

The following support case has sosreport and must-gather from a cluster experiencing this issue.
https://access.redhat.com/support/cases/#/case/02649417


From syslog on the worker node

May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: E0521 15:45:06.047028    1549 mount_linux.go:140] Mount failed: exit status 32
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files --scope -- mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/share-1 /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: mount error(2): No such file or directory
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: E0521 15:45:06.047166    1549 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/azure-file/d36100fb-6ba5-48c5-bd03-b114aa8878f3-azure-files\" (\"d36100fb-6ba5-48c5-bd03-b114aa8878f3\")" failed. No retries permitted until 2020-05-21 15:47:08.047135184 +0000 UTC m=+1804551.819088583 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"azure-files\" (UniqueName: \"kubernetes.io/azure-file/d36100fb-6ba5-48c5-bd03-b114aa8878f3-azure-files\") pod \"www\" (UID: \"d36100fb-6ba5-48c5-bd03-b114aa8878f3\") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files --scope -- mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/share-1 /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files\nOutput: Running scope as unit: run-r40d29acf1bca46ad9d9e9f30c0a379b2.scope\nmount error(2): No such file or directory\nRefer to the mount.cifs(8) manual page (e.g. man mount.cifs)\n"
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: I0521 15:45:06.047213    1549 event.go:255] Event(v1.ObjectReference{Kind:"Pod", Namespace:"nick-test", Name:"www", UID:"d36100fb-6ba5-48c5-bd03-b114aa8878f3", APIVersion:"v1", ResourceVersion:"22343333", FieldPath:""}): type: 'Warning' reason: 'FailedMount' MountVolume.SetUp failed for volume "azure-files" : mount failed: exit status 32
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files --scope -- mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/share-1 /var/lib/kubelet/pods/d36100fb-6ba5-48c5-bd03-b114aa8878f3/volumes/kubernetes.io~azure-file/azure-files
May 21 15:45:06 ocpnonprod02azcentralusworkera3 hyperkube[1549]: mount error(2): No such file or directory



From event stream:


Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/af96c7a9-3922-4ebe-81e2-69e6154bd342/volumes/kubernetes.io~azure-file/azure-files --scope -- mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/ocpshare /var/lib/kubelet/pods/af96c7a9-3922-4ebe-81e2-69e6154bd342/volumes/kubernetes.io~azure-file/azure-files
Output: Running scope as unit: run-r2d93d372e7464176ad7c1e4a8a73442e.scope
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
20s         Warning   FailedMount          pod/www                            MountVolume.SetUp failed for volume "azure-files" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/af96c7a9-3922-4ebe-81e2-69e6154bd342/volumes/kubernetes.io~azure-file/azure-files --scope -- mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/ocpshare /var/lib/kubelet/pods/af96c7a9-3922-4ebe-81e2-69e6154bd342/volumes/kubernetes.io~azure-file/azure-files
Output: Running scope as unit: run-rab06e13fa55d47d08efb9bb4e0244e43.scope
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
1s          Warning   FailedMount          pod/www                            Unable to attach or mount volumes: unmounted volumes=[www-persistent-storage], unattached volumes=[www-persistent-storage default-token-4vhgz]: timed out waiting for the condition




Deployed Objects:

apiVersion: v1
kind: List
items:
- apiVersion: v1
  data:
    azurestorageaccountkey: <base64key>
    azurestorageaccountname: <base64name>
  kind: Secret
  metadata:
    name: azure-files
    namespace: nick-test
  type: Opaque

- apiVersion: v1
  kind: Pod
  metadata:
    name: www
    labels:
      name: www
  spec:
    containers:
    - name: www
      image: nginx:alpine
      ports:
      - containerPort: 80
        name: www
      volumeMounts:
      - name: www-persistent-storage
        mountPath: /usr/share/nginx/html
    volumes:
    - name: www-persistent-storage
      persistentVolumeClaim:
        claimName: http-claim

- kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: http-claim
  spec:
    accessModes:
    - ReadWriteOnce
    storageClassName: azure-files
    resources:
      requests:
        storage: 3Gi

- apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: azure-files
  spec:
    capacity:
      storage: 5Gi
    accessModes:
    - ReadWriteOnce
    storageClassName: azure-files
    azureFile:
      secretName: azure-files
      shareName: ocpshare
      readOnly: false

I have verified the share name matches whats in the PV.

In Azure:
https://mystorageaccount8022.file.core.windows.net/ocpshare

In the PV:
    azureFile:
      secretName: azure-files
      secretNamespace: nick-test
      shareName: ocpshare
      readOnly: false



I have also tried mounting the share manually from a worker node and seeing the same error:

[root@ocpnonprod02azcentralusworkera2 /]# mkdir /tmp/test
[root@ocpnonprod02azcentralusworkera2 /]# mount -t cifs -o username=mystorageaccount8022,password=<pass>,file_mode=0777,dir_mode=0777,vers=3.0 //mystorageaccount8022.file.core.windows.net/ocpshare /tmp/test

Sorry didn't post full command:

[root@ocpnonprod02azcentralusworkera2 /]# mkdir /tmp/test
[root@ocpnonprod02azcentralusworkera2 /]# mount -t cifs -o username=mystorageaccount8022,password=Y1g
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
[root@ocpnonprod02azcentralusworkera2 /]# mount -t cifs -o username=mystorageaccount8022,password=Y1g
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Sorry, I did not realize there already is must-gather & sosreport.

May 21 15:36:05 ocpnonprod02azcentralusworkera3 kernel: CIFS: Attempting to mount //mystorageaccount8022.file.core.windows.net/share-1
May 21 15:36:15 ocpnonprod02azcentralusworkera3 kernel: CIFS VFS: Error connecting to socket. Aborting operation.

Not sure how it relates "No such file or directory" error in userspace, 10 seconds in between these two messages is quite a lot even for Azure. Debugging suggestions from comment #5 still apply - is the mount command that OCP used correct? Can you please check the username, password and share address with Azure? Looking at the file share, press "Connect" / "Linux" and check the mount command used. They don't pass username / password as mount options, but via a file, but that's negligible.

This may be an issue with the OpenShift nodes not being able to reach the Azure Files shares.

Let me set up private endpoints in the subnet that these worker nodes are in and test. 

I will report back.

Nick, any news from your networking investigation? Is there a bug on OCP side?

Have not yet been able to test deploying against an Azure private endpoint, waiting on customer networking teams.

If Red Hat has verified deploying NFS Persistent Volumes against Azure Storage Accounts with Private Endpoints (instead of against a public ip address), we can close this bug and I can reopen it when I am able to test if there is an issue.

Otherwise we can leave this open and I can update it when I am able to test.

It may be that we just need to update the OpenShift documentation to include information about configuring Azure Files in private environments.

Qin, do you tests Azure in disconnected environment? Did you have any issue with Azure File? And did you use Private Endpoints? If not, would it be possible to test it?

Hello Jan,

We tested azure file in the disconnected + private azure cluster, it works well.

I don't know if this is a cluster used azure private endpoints.

I have not had the ability to test in client environment, but its very likely the missing private endpoint was the issue. Closing.