Bug 1567610

Summary: Storage Class parameters ignored after upgrade to 3.7.42
Product: OpenShift Container Platform Reporter: hgomes
Component: StorageAssignee: hchen
Status: CLOSED WORKSFORME QA Contact: Wenqi He <wehe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, aos-storage-staff, wehe, wmeng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-23 14:08:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description hgomes 2018-04-15 12:34:43 UTC
Description of problem:

Upgrade cluster to 3.7.42 and dynamically provisioned persistent volumes stopped working with this error message:

atomic-openshift-master-controllers:  pv_controller.go:1380] failed to provision volume for claim "projectname/pvcname" with StorageClass "generic": Create Storage Account: pvc3096215148001, error: storage.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="LocationRequired" Message="The location property is required for this definition."

Storage class definition BEFORE the upgrade:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: "generic"
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
    volume.beta.kubernetes.io/storage-class: "generic"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
provisioner: kubernetes.io/azure-disk
parameters:
  storageAccount: <account-name>

File /etc/azure/azure.conf BEFORE the upgrade, it had definitions:
resourceGroup, subscriptionID, aadClientID,aadClientSecret,tenantID,useIntanceMedatadata

The storage account exists in Azure and contains dynamically created persistent volumes.

How reproducible:
Before the configuration changes, it was able to reproduce this behavior every time I created a persistent volume claim with storage class "generic".


Additional info:

Comment 1 hgomes 2018-04-15 12:41:31 UTC
As a workaround:

Added two parameters (location and skuName) in the storage class:

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: "generic"
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
    volume.beta.kubernetes.io/storage-class: "generic"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
provisioner: kubernetes.io/azure-disk
parameters:
  location: westus
  storageAccount: <storage-name> <<<<<
  skuName: Premium_LRS           <<<<<

Added the 'location' in /etc/azure/azure.conf:

{"resourceGroup": "lab-purple-edition", "subscriptionID": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "aadClientID": "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", "aadClientSecret": "SECRET", "tenantID": "BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB", "useInstanceMetadata": true, "location": "purple" }

Comment 2 Wenqi He 2018-04-16 03:28:13 UTC
Please refer my upstream issue: https://github.com/kubernetes/kubernetes/issues/61265 

The error message shows your azure conf need the "location" parameter. 

After kube 1.7 azure storage class has been revised, you need to specify the "kind: Dedicated" if you still want to use the azure storage class you created before upgrade, otherwise, add location in you azure conf to use the default "kind: Shared" and it will create few new storage accounts and keep using the new created ones.

Comment 3 Wenqi He 2018-04-16 03:39:13 UTC
Example of the sc:
$ cat desc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: addsc 
provisioner: kubernetes.io/azure-disk
parameters:
  kind: Dedicated
  storageaccount: eastusimg

Comment 4 hchen 2018-04-23 14:08:56 UTC
Please refer to Comment 3 for resolution.