Description of problem: Dynamically provision a Cinder PV, the Cinder volume can not be created. Error "Error creating cinder volume: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead" Version-Release number of selected component (if applicable): openshift v3.7.0-0.125.0 kubernetes v1.7.0+695f48a16f etcd 3.2.1 How reproducible: Always Steps to Reproduce: 1. Setup OCP with standalone api and controller services. 2. Create a default storageclass apiVersion: v1 items: - apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: 2017-09-12T07:27:11Z name: generic namespace: "" resourceVersion: "24496" selfLink: /apis/storage.k8s.io/v1/storageclasses/generic uid: caf75660-978b-11e7-8799-fa163e765c12 provisioner: kubernetes.io/cinder kind: List metadata: resourceVersion: "" selfLink: "" 3. Create a PVC apiVersion: v1 items: - apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: openshift.io/generated-by: OpenShiftNewApp volume.alpha.kubernetes.io/storage-class: foo volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder creationTimestamp: 2017-09-12T07:30:01Z labels: app: mysql-persistent template: mysql-persistent-template name: mysql namespace: wzheng4 resourceVersion: "24800" selfLink: /api/v1/namespaces/wzheng4/persistentvolumeclaims/mysql uid: 2feb4297-978c-11e7-8799-fa163e765c12 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: generic status: phase: Pending kind: List metadata: resourceVersion: "" selfLink: "" Actual results: PV wasn't provisioned. Name: mysql Namespace: wzheng4 StorageClass: generic Status: Pending Volume: Labels: app=mysql-persistent template=mysql-persistent-template Annotations: openshift.io/generated-by=OpenShiftNewApp volume.alpha.kubernetes.io/storage-class=foo volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder Capacity: Access Modes: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 21m 1m 83 persistentvolume-controller Warning ProvisioningFailed Failed to provision volume with StorageClass "generic": Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]} Expected results: PVs are provisioned successfully. Master Log: Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]} Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: E0912 03:49:46.734186 18334 openstack.go:615] Error when retrieving list of supported blockstorage api versions: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]} Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: E0912 03:49:46.734199 18334 openstack_volumes.go:305] Unable to initialize cinder client for region: regionOne Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: I0912 03:49:46.734203 18334 cinder_util.go:207] Error creating cinder volume: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]} Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: I0912 03:49:46.734219 18334 pv_controller.go:1331] failed to provision volume for claim "wzheng4/mysql" with StorageClass "generic": Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead ... Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.460737 73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.460846 73893 wrap.go:42] GET /api/v1/nodes: (20.271503ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798] Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461142 73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461223 73893 wrap.go:42] GET /api/v1/nodes: (20.340152ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798] Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461329 73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461405 73893 wrap.go:42] GET /api/v1/nodes: (20.276866ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798] Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462865 18334 cinder_util.go:149] Error listing nodes Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462887 18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462903 18334 pv_controller.go:1331] failed to provision volume for claim "wzheng4/mysql" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462981 18334 cinder_util.go:149] Error listing nodes Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462988 18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462998 18334 pv_controller.go:1331] failed to provision volume for claim "wzheng3/redis" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463058 18334 cinder_util.go:149] Error listing nodes Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463065 18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463074 18334 pv_controller.go:1331] failed to provision volume for claim "wzheng2/redis" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster Additional info: Attached master log
I assume that the OpenShift cloud.conf file has Cinder version automatic detection configured. Such configuration looks like: [BlockStorage] bs-version = auto Please, can you confirm this?
There's a workaround for this bug: configure in cloud.conf a specific Cinder version, e.g. V2: [BlockStorage] bs-version = v2 IMO, the problem is that in case OpenShift is configured to auto-detect Cider version it uses Cinder V1 to query for available Cinder versions, however, as the Cinder V1 is deprecated in the used OpenStack cluster the autodetection fails.
@ppospisi Thank you, that fixed the problem! I'll transfer its component to docs to have this configuration item documented.
Sorry move back to storage to decide if we want to fix it first.
I looked into the issue again an found out that the issue is fixed in gophercloud library [1] and in Kubernetes [2] for K8s 1.8 release. So upgrading the gophercloud library to commit [3] in OpenShift vendor directory should fix the problem. [1] https://github.com/gophercloud/gophercloud/pull/383 [2] https://github.com/kubernetes/kubernetes/pull/47602 [3] https://github.com/gophercloud/gophercloud/commit/ed590d9afe113c6107cd60717b196155e6579e78
I backported the gophercloud library fix [1] and created PR [2]. [1] https://github.com/gophercloud/gophercloud/pull/383 [2] https://github.com/openshift/origin/pull/16321
The PR https://github.com/openshift/origin/pull/16321 was merged.
Verified this is fixed in 3.7.0-0.127. Cinder volume provisioned successfully without the additional config from comment 1.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188
I just hit this on a brand new OCP 3.7 plus OSP 12. Did this with advanced install so not sure where to update cloud.conf with bs-version=v2 as i installed with openshift-ansible pointing at an inventory file. Can I add bs-version=v2 to the inventory at install time? Does it go under a subsection? [BlockStorage] ?
Please refer to this doc [1]. Add to cloud.conf with the following: [BlockStorage] bs-version = v2 1. https://docs.openshift.com/container-platform/3.6/install_config/configuring_openstack.html#configuring-openstack-variables
With an advanced install of OCP 3.7 there is no cloud.conf. So you need this: https://github.com/openshift/openshift-ansible/pull/6611/files Basically you need to set the ansible variable: openshift_cloudprovider_openstack_blockstorage_version but it won't work without a change to the jinja2 template as well roles/openshift_cloud_provider/templates/openstack.conf.j2 See link. Also the link you shared doesn't mention the value required, so could be confusing.