Bug 1457092 - [3.6][Cinder] Dynamic provision failed when zone is not specified in the StorageClass
Summary: [3.6][Cinder] Dynamic provision failed when zone is not specified in the Stor...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: Pavel Pospisil
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks: 1447568
TreeView+ depends on / blocked
 
Reported: 2017-05-31 06:22 UTC by Jianwei Hou
Modified: 2017-11-28 21:56 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-11-28 21:56:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Jianwei Hou 2017-05-31 06:22:15 UTC
Description of problem:
PV can not be provisioned when the zone is not set in the storageclass. 
PV can not be provisioned using 'alpha' annotation.

Version-Release number of selected component (if applicable):
openshift v3.6.86
kubernetes v1.6.1+5115d708d7
etcd 3.1.0

How reproducible:
Always

Steps to Reproduce:
1. Create a PVC with alpha annotation, which should dynamically provision a PV.
{
  "kind": "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "c1",
    "labels": {
         "name": "testing"
     },
    "annotations": {
        "volume.alpha.kubernetes.io/storage-class": "foo"
    }
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "1Gi"
      }
    }
  }
}

2. Create a PVC using StorageClass, the StorageClass does not have any parameters. Expecting a PV provisioned in the 'nova' availability zone.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cinder1
provisioner: kubernetes.io/cinder

{
  "kind": "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "cinderc1",
    "annotations": {
        "volume.beta.kubernetes.io/storage-class": "cinder1"
    }
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "1Gi"
      }
    }
  }
}


3. Create a PVC using a StorageClass that has availability zone parameter.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cinder2
provisioner: kubernetes.io/cinder
parameters:
  availability: nova


Actual results:
1.  PV can not be provisioned
# oc describe pvc c1
Name:		c1
Namespace:	jhou
StorageClass:	foo
Status:		Pending
Volume:		
Labels:		name=testing
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  5m		3s		22	{persistent-volume-controller }			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "": User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot list all nodes in the cluster

2. Same failure and info as 1.

3. PV successfully provisioned


Expected results:
Cinder PV should be provisioned using alpha annotation.
Cinder PV should be provisioned when the storageclass does not set an availability zone.

Master Log:
```
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.393102   38295 wrap.go:42] GET /api/v1/nodes: (9.763061ms) 403 [[openshift/v1.6.1+5115d708d7 (linux/amd64) kubernetes/5115d70 system:serviceaccount:openshift-infra:pv-binder-controller] 10.8.175.59:53646]
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.394081   38295 cinder_util.go:148] Error listing nodes
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.394090   38295 cinder_util.go:194] error getting zone information: User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot list all nodes in the cluster
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.394100   38295 pv_controller.go:1312] failed to provision volume for claim "default/platform-pvc" with StorageClass "": User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot list all nodes in the cluster
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.396467   38295 wrap.go:42] GET /api/v1/nodes: (7.571212ms) 403 [[openshift/v1.6.1+5115d708d7 (linux/amd64) kubernetes/5115d70 system:serviceaccount:openshift-infra:pv-binder-controller] 10.8.175.59:53646]
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.396921   38295 wrap.go:42] PATCH /api/v1/namespaces/jhou/events/c1.14c39b3b29a8128e: (5.276618ms) 200 [[openshift/v1.6.1+5115d708d7 (linux/amd64) kubernetes/5115d70] 10.8.175.59:53592]
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.397178   38295 cinder_util.go:148] Error listing nodes
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.397187   38295 cinder_util.go:194] error getting zone information: User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot list all nodes in the cluster
May 31 02:08:00 host-8-175-59 atomic-openshift-master: I0531 02:08:00.397203   38295 pv_controller.go:1312] failed to provision volume for claim "jhou/cinderc1" with StorageClass "cinder1": User "system:serviceaccount:openshift-infra:pv-binder-controller" cannot list all nodes in the cluster
```

Additional info:
Deploy same version OCP on EC2, this is not reproduced, so this might be caused by OpenStack availability zone.

[root@host-8-175-59 ~]# oadm policy who-can list node
Namespace: default
Verb:      list
Resource:  nodes

Users:  jhou
        system:admin
        system:kube-controller-manager
        system:kube-scheduler
        system:master
        system:serviceaccount:default:router
        system:serviceaccount:kube-system:attachdetach-controller
        system:serviceaccount:kube-system:daemon-set-controller
        system:serviceaccount:kube-system:generic-garbage-collector
        system:serviceaccount:kube-system:namespace-controller
        system:serviceaccount:kube-system:node-controller
        system:serviceaccount:kube-system:pod-garbage-collector
        system:serviceaccount:kube-system:resourcequota-controller
        system:serviceaccount:kube-system:route-controller
        system:serviceaccount:kube-system:service-controller
        system:serviceaccount:kube-system:ttl-controller
        system:serviceaccount:management-infra:management-admin
        system:serviceaccount:openshift-infra:pv-attach-detach-controller
        system:serviceaccount:openshift-infra:service-load-balancer-controller

Groups: system:cluster-admins
        system:cluster-readers
        system:masters
        system:node-admins
        system:nodes

[root@host-8-175-59 ~]# oc get clusterrolebindings|grep pv-binder-controller
system:pv-binder-controller                                 /system:pv-binder-controller                                                                                                                 openshift-infra/pv-binder-controller

Comment 6 Jianwei Hou 2017-07-05 05:22:24 UTC
Verified in openshift v3.6.131, both issues have been fixed.

Comment 10 errata-xmlrpc 2017-11-28 21:56:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.