Description of problem: When OpenShift is deployed in Azure regional clouds (Azure Germany, China, US Gov and others), Azure File can't use for PV. We describe Azure regional clouds are supported. https://access.redhat.com/articles/product-configuration-for-azure Version-Release number of selected component (if applicable): kubernetes v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1. Deploy OpenShift on Azure Germany (I confirmed on Azure Germany, but I think it happens on other Azure Regional Clouds (US Gov, China)) 2. Set up Azure File as a Persistent Volume https://docs.openshift.com/container-platform/3.6/install_config/persistent_storage/persistent_storage_azure_file.html ~~~ $ cat azure-secret-file.yaml apiVersion: v1 kind: Secret metadata: name: azure-secret-file type: Opaque stringData: azurestorageaccountname: ocpsa azurestorageaccountkey: <key> $ oc create -f azure-secret-file.yaml $ cat pv-fileshare1.yaml apiVersion: "v1" kind: "PersistentVolume" metadata: name: "pv-azureshare" spec: capacity: storage: "10Gi" accessModes: - "ReadWriteMany" azureFile: secretName: azure-secret-file shareName: fileshare1 readOnly: false $ oc create -f pv-fileshare1.yaml $ cat pvc-fileshare1.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-fileshare1 spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi volumeName: pv-azureshare $ oc create -f pvc-fileshare1.yaml ~~~ 3. Deploy a pod with this PVC. Actual results: A pod is failed to deploy. in the node log: ~~~ Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: I1115 22:29:03.807900 16090 reconciler.go:318] MountVolume operation started for volume "kubernetes.io/azure-file/382ecff0-ca7e-11e7-ad78-0017fa104c4b-pv-azureshare" (spec.Name: "pv-azureshare") to pod "382ecff0-ca7e-11e7-ad78-0017fa104c4b" (UID: "382ecff0-ca7e-11e7-ad78-0017fa104c4b"). Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: E1115 22:29:03.823172 16090 mount_linux.go:164] Mount failed: exit status 1 Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Mounting command: systemd-run Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Mounting arguments: --description=Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/382ecff0-ca7e-11e7-ad78-0017fa104c4b/volumes/kubernetes.io~azure-file/pv-azureshare --scope -- mount -t cifs -o vers=3.0,username=ocpsa,password=5nFDkCxxNBQLQehCNLIJ481WiENHIX6uWZfFsQw/aMGx4ob1hVPHMkwCoH31N/zlad2NHjKQXKxukOLpjalvHw==,dir_mode=0777,file_mode=0777 //ocpsa.file.core.windows.net/fileshare1 /var/lib/origin/openshift.local.volumes/pods/382ecff0-ca7e-11e7-ad78-0017fa104c4b/volumes/kubernetes.io~azure-file/pv-azureshare Nov 15 22:29:03 tatanaka-ocp-node1 atomic-openshift-node[16090]: Output: Running scope as unit run-53531.scope. ~~~ Expected results: Azure File is mounted to a pod and the pod becomes running. Additional info: This is because Azure regional cloud has different endpoints. https://access.redhat.com/articles/product-configuration-for-azure However, the endpoint is hard coded. https://github.com/openshift/ose/blob/enterprise-3.6/vendor/k8s.io/kubernetes/pkg/volume/azure_file/azure_file.go#L209 It appears this has not been fixed even at 3.7. https://github.com/openshift/ose/blob/enterprise-3.7/vendor/k8s.io/kubernetes/pkg/volume/azure_file/azure_file.go#L210 Kubernetes 1.8 has fixed the issue. https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/volume/azure_file/azure_file.go For documentation bug: https://bugzilla.redhat.com/show_bug.cgi?id=1506880 Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
fix is bundled at https://github.com/openshift/origin/pull/17052
Test on below version: openshift v3.9.0-0.22.0 kubernetes v1.9.1+a0ce1bc657 The azure file failed to mount. Please track in bug #1536362 @Takayoshi Hi, seems my azure account cannot create vm on Germany, China, and the countries you mentioned. Could you please help me verify this bug when it is ready next time? Otherwise I will choose a very less used region to verify this but not the one you mentioned. Thanks.
move to ON_QA and wait for verification
Hi Wenqi, I still have my valid account for Azure Germany. I can test it with this account. Or, if you want your account, I'll ask Microsoft engineer. This account is provided by Microsoft engineer for testing purpose.
Hi Takayoshi, Yes pls help to verify it. I think we can verify this on latest 3.7 firstly as the customer request. For 3.9, it blocks by bug #1536362, all the Azure file could not be used from my test. We track that later. I know there is installation blocker bug for Azure, and I heard Glenn has work around and also you tried it successfully? Thanks
Tested on below version: openshift v3.9.3 kubernetes v1.9.1+a0ce1bc657 Azure file works well on Germany region both with 3.7 and 3.9: $ oc get pv pv-azureshare1 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/bound-by-controller: "yes" creationTimestamp: 2018-03-08T06:45:04Z name: pv-azureshare1 resourceVersion: "3456456" selfLink: /api/v1/persistentvolumes/pv-azureshare1 uid: 3b66aea3-229c-11e8-8d0a-0017fa104c4b spec: accessModes: - ReadWriteMany azureFile: secretName: azure-secret-file secretNamespace: null shareName: fileshare1 capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: pvc-fileshare namespace: test resourceVersion: "3456454" uid: c84a22ea-229c-11e8-8d0a-0017fa104c4b persistentVolumeReclaimPolicy: Retain status: phase: Bound $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-fileshare Bound pv-azureshare1 1Gi RWX 5m $ oc get pods NAME READY STATUS RESTARTS AGE azure 1/1 Running 0 5m $ cat pod.yaml apiVersion: v1 kind: Pod metadata: name: azure spec: containers: - name: test-pod image: centos command: - "/bin/sh" args: - "-c" - "sleep 3600000" volumeMounts: - name: azure-pvc mountPath: /mnt/azure volumes: - name: azure-pvc persistentVolumeClaim: claimName: pvc-fileshare
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489