Bug 1513859

Summary: Failed to mount Azure File as a Persistent Volume in Azure regional cloud (Germany and so on)
Product: OpenShift Container Platform Reporter: Takayoshi Tanaka <tatanaka>
Component: StorageAssignee: hchen
Status: CLOSED ERRATA QA Contact: Wenqi He <wehe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: aos-bugs, aos-storage-staff, bchilds, eparis, florian.herzog, gchakkar, hchen, mtaru, pdwyer, tatanaka, wehe, xtian
Target Milestone: ---   
Target Release: 3.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1532809 (view as bug list) Environment:
Last Closed: 2018-03-28 14:12:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1532809    

Description Takayoshi Tanaka 2017-11-16 06:37:43 UTC
Description of problem:
When OpenShift is deployed in Azure regional clouds (Azure Germany, China, US Gov and others), Azure File can't use for PV.
We describe Azure regional clouds are supported.
https://access.redhat.com/articles/product-configuration-for-azure

Version-Release number of selected component (if applicable):
kubernetes v1.6.1+5115d708d7

How reproducible:
Always

Steps to Reproduce:
1. Deploy OpenShift on Azure Germany (I confirmed on Azure Germany, but I think it happens on other Azure Regional Clouds (US Gov, China))
2. Set up Azure File as a Persistent Volume
https://docs.openshift.com/container-platform/3.6/install_config/persistent_storage/persistent_storage_azure_file.html

~~~
$ cat azure-secret-file.yaml 
apiVersion: v1
kind: Secret
metadata:
  name: azure-secret-file
type: Opaque
stringData:
  azurestorageaccountname: ocpsa
  azurestorageaccountkey: <key>

$ oc create -f azure-secret-file.yaml 

$ cat pv-fileshare1.yaml 
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "pv-azureshare" 
spec:
  capacity:
    storage: "10Gi" 
  accessModes:
    - "ReadWriteMany"
  azureFile: 
    secretName: azure-secret-file
    shareName: fileshare1
    readOnly: false 

$ oc create -f pv-fileshare1.yaml 

$ cat pvc-fileshare1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-fileshare1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-azureshare

$ oc create -f pvc-fileshare1.yaml
~~~

3. Deploy a pod with this PVC.

Actual results:
A pod is failed to deploy.

in the node log:
~~~
Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: I1115 22:29:03.807900   16090 reconciler.go:318] MountVolume operation started for volume "kubernetes.io/azure-file/382ecff0-ca7e-11e7-ad78-0017fa104c4b-pv-azureshare" (spec.Name: "pv-azureshare") to pod "382ecff0-ca7e-11e7-ad78-0017fa104c4b" (UID: "382ecff0-ca7e-11e7-ad78-0017fa104c4b").
Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: E1115 22:29:03.823172   16090 mount_linux.go:164] Mount failed: exit status 1
Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Mounting command: systemd-run
Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/382ecff0-ca7e-11e7-ad78-0017fa104c4b/volumes/kubernetes.io~azure-file/pv-azureshare --scope -- mount -t cifs -o vers=3.0,username=ocpsa,password=5nFDkCxxNBQLQehCNLIJ481WiENHIX6uWZfFsQw/aMGx4ob1hVPHMkwCoH31N/zlad2NHjKQXKxukOLpjalvHw==,dir_mode=0777,file_mode=0777 //ocpsa.file.core.windows.net/fileshare1 /var/lib/origin/openshift.local.volumes/pods/382ecff0-ca7e-11e7-ad78-0017fa104c4b/volumes/kubernetes.io~azure-file/pv-azureshare
Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Output: Running scope as unit run-53531.scope.
~~~

Expected results:
Azure File is mounted to a pod and the pod becomes running.

Additional info:
This is because Azure regional cloud has different endpoints.
https://access.redhat.com/articles/product-configuration-for-azure

However, the endpoint is hard coded.
https://github.com/openshift/ose/blob/enterprise-3.6/vendor/k8s.io/kubernetes/pkg/volume/azure_file/azure_file.go#L209

It appears this has not been fixed even at 3.7.
https://github.com/openshift/ose/blob/enterprise-3.7/vendor/k8s.io/kubernetes/pkg/volume/azure_file/azure_file.go#L210

Kubernetes 1.8 has fixed the issue.
https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/volume/azure_file/azure_file.go

For documentation bug: https://bugzilla.redhat.com/show_bug.cgi?id=1506880

Description of problem:

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 2 hchen 2017-11-17 13:15:12 UTC
fix is bundled at https://github.com/openshift/origin/pull/17052

Comment 7 Wenqi He 2018-01-23 10:24:28 UTC
Test on below version:
openshift v3.9.0-0.22.0
kubernetes v1.9.1+a0ce1bc657

The azure file failed to mount. Please track in bug #1536362

@Takayoshi Hi, seems my azure account cannot create vm on Germany, China, and the countries you mentioned. Could you please help me verify this bug when it is ready next time? Otherwise I will choose a very less used region to verify this but not the one you mentioned. Thanks.

Comment 8 hchen 2018-01-23 16:22:10 UTC
move to ON_QA and wait for verification

Comment 9 Takayoshi Tanaka 2018-01-25 01:49:26 UTC
Hi Wenqi,

I still have my valid account for Azure Germany. I can test it with this account. Or, if you want your account, I'll ask Microsoft engineer. This account is provided by Microsoft engineer for testing purpose.

Comment 10 Wenqi He 2018-01-26 08:37:49 UTC
Hi Takayoshi,
Yes pls help to verify it. I think we can verify this on latest 3.7 firstly as the customer request. For 3.9, it blocks by bug #1536362, all the Azure file could not be used from my test. We track that later.
I know there is installation blocker bug for Azure, and I heard Glenn has work around and also you tried it successfully? Thanks

Comment 22 Wenqi He 2018-03-08 06:57:06 UTC
Tested on below version:
openshift v3.9.3
kubernetes v1.9.1+a0ce1bc657

Azure file works well on Germany region both with 3.7 and 3.9:

$ oc get pv pv-azureshare1 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: 2018-03-08T06:45:04Z
  name: pv-azureshare1
  resourceVersion: "3456456"
  selfLink: /api/v1/persistentvolumes/pv-azureshare1
  uid: 3b66aea3-229c-11e8-8d0a-0017fa104c4b
spec:
  accessModes:
  - ReadWriteMany
  azureFile:
    secretName: azure-secret-file
    secretNamespace: null
    shareName: fileshare1
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-fileshare
    namespace: test
    resourceVersion: "3456454"
    uid: c84a22ea-229c-11e8-8d0a-0017fa104c4b
  persistentVolumeReclaimPolicy: Retain
status:
  phase: Bound


$ oc get pvc
NAME             STATUS    VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-fileshare    Bound     pv-azureshare1   1Gi        RWX                           5m

$ oc get pods
NAME                  READY     STATUS    RESTARTS   AGE
azure                 1/1       Running   0          5m

$ cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
 name: azure
spec:
 containers:
  - name: test-pod
    image: centos
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "sleep 3600000"
    volumeMounts:
      - name: azure-pvc
        mountPath: /mnt/azure
 volumes:
      - name: azure-pvc
        persistentVolumeClaim:
           claimName: pvc-fileshare

Comment 25 errata-xmlrpc 2018-03-28 14:12:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489