Bug 1500247 - Azure disk: Failed to provision if storage class has no parameters
Summary: Azure disk: Failed to provision if storage class has no parameters
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: hchen
QA Contact: Wenqi He
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-10 09:29 UTC by Wenqi He
Modified: 2017-10-26 12:56 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-10-26 12:56:15 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Wenqi He 2017-10-10 09:29:11 UTC
Description of problem:
Create a storage class with no parameters, dynamic provision failed

Version-Release number of selected component (if applicable):
openshift v3.7.0-0.146.0
kubernetes v1.7.6+a08f5eeb62

How reproducible:
Always

Steps to Reproduce:
1. Create a storage class with no parameters
2. Create a pvc which point this storage class
3. PVC would keep pending

Actual results:
$ oc get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
azpvc     Pending                                      sc-kytwu       4m

$ oc describe pvc azpvc
Name:		azpvc
Namespace:	kytwu
StorageClass:	sc-kytwu
Status:		Pending
Volume:		
Labels:		<none>
Annotations:	volume.beta.kubernetes.io/storage-class=sc-kytwu
		volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  5m		5s		22	persistentvolume-controller			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "sc-kytwu": Create Storage Account: p443172070, error: storage.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="LocationRequired" Message="The location property is required for this definition."

Expected results:
PVC can be provisioned and bound

Master Log:
Oct 10 09:10:42 wehe-master-etcd-nfs-1 atomic-openshift-master-controllers: I1010 09:10:42.354472   21378 pv_controller.go:1337] failed to provision volume for claim "default/registry-pvc" with StorageClass "grs": azureDisk - Standard_GRS is not supported sku/storageaccounttype. Supported values are [Premium_LRS Standard_LRS]
Oct 10 09:10:44 wehe-master-etcd-nfs-1 atomic-openshift-node: I1010 09:10:44.225558   14737 kubelet_node_status.go:490] Using Node Hostname from cloudprovider: "wehe-master-etcd-nfs-1.eastus.cloudapp.azure.com"
Oct 10 09:10:44 wehe-master-etcd-nfs-1 atomic-openshift-master-api: I1010 09:10:44.240381   21352 rest.go:349] Starting watch for /api/v1/services, rv=2299 labels= fields= timeout=7m33s
Oct 10 09:10:49 wehe-master-etcd-nfs-1 atomic-openshift-master-controllers: I1010 09:10:49.692346   21378 azure_blobDiskController.go:602] azureDisk - Creating storage account p443172070 type Standard_LRS
Oct 10 09:10:49 wehe-master-etcd-nfs-1 atomic-openshift-master-controllers: I1010 09:10:49.733551   21378 pv_controller.go:1337] failed to provision volume for claim "kytwu/azpvc" with StorageClass "sc-kytwu": Create Storage Account: p443172070, error: storage.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="LocationRequired" Message="The location property is required for this definition."
Oct 10 09:10:50 wehe-master-etcd-nfs-1 atomic-openshift-master-api: I1010 09:10:50.478416   21352 rest.go:349] Starting watch for /apis/network.openshift.io/v1/egressnetworkpolicies, rv=2254 labels= fields= timeout=6m6s
Oct 10 09:10:54 wehe-master-etcd-nfs-1 atomic-openshift-node: I1010 09:10:54.409112   14737 kubelet_node_status.go:490] Using Node Hostname from cloudprovider: "wehe-master-etcd-nfs-1.eastus.cloudapp.azure.com"
Oct 10 09:10:56 wehe-master-etcd-nfs-1 atomic-openshift-master-api: I1010 09:10:56.268606   21352 rest.go:349] Starting watch for /apis/extensions/v1beta1/networkpolicies, rv=19624 labels= fields= timeout=6m16s
Oct 10 09:10:56 wehe-master-etcd-nfs-1 atomic-openshift-master-api: I1010 09:10:56.652807   21352 rest.go:349] Starting watch for /apis/extensions/v1beta1/daemonsets, rv=19695 labels= fields= timeout=6m7s
Oct 10 09:10:56 wehe-master-etcd-nfs-1 atomic-openshift-node: I1010 09:10:56.905512   14737 qos_container_manager_linux.go:286] [ContainerManager]: Updated QoS cgroup configuration


Node Log (of failed PODs):

PV Dump:

PVC Dump:
$ oc export pvc azpvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-class: sc-kytwu
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
  creationTimestamp: null
  name: azpvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
status: {}

StorageClass Dump (if StorageClass used by PV/PVC):
# oc export storageclass sc-kytwu
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  name: sc-kytwu
provisioner: kubernetes.io/azure-disk

Additional info:
From the master log, we can see another issue, sku name only support Premium_LRS Standard_LRS now, not sure whether this is new design for OCP

Comment 6 hchen 2017-10-11 13:42:33 UTC
Turns out the azure disk refactoring in 1.7 introduced a regression. It requires "location" to be set in azure cloud config.

Comment 10 hchen 2017-10-24 14:19:28 UTC
Upstream issue
https://github.com/kubernetes/kubernetes/issues/52396

Comment 11 hchen 2017-10-24 14:25:04 UTC
for workaround, please set storage account name in the storage class.

Comment 12 hchen 2017-10-25 15:08:04 UTC
There is another issue with running azure provisioner in openshift, https://github.com/kubernetes/kubernetes/issues/54570

Comment 13 hchen 2017-10-25 15:09:27 UTC
Due to kubernetes issues #54570 and #52396, we cannot use "Kind" parameter (will hit #54570) or empty parameter (will hit #52396)in openshift.

Comment 16 hchen 2017-10-26 12:54:32 UTC
Thanks, both location and storageAccount are needed for 3.7.

Comment 17 hchen 2017-10-26 12:54:45 UTC
Thanks, both location and storageAccount are needed for 3.7.


Note You need to log in before you can comment on or make changes to this bug.