Bug 1490768

Summary: Failed to provision cinder volume, got http response code 300
Product: OpenShift Container Platform Reporter: Jianwei Hou <jhou>
Component: StorageAssignee: Pavel Pospisil <ppospisi>
Status: CLOSED ERRATA QA Contact: Jianwei Hou <jhou>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: aos-bugs, aos-storage-staff, asimonel, bchilds, hchen, jhou, jokerman, lxia, mmccomas, ppospisi, xtian
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Gophercloud library used by OpenShift to communicate with OpenStack API does not accept HTTP status 300 in pagination. Consequence: It is not possible to dynamically provision Cinder persistent volumes. Error "Error creating cinder volume: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead" is reported. Fix: Gophercloud library accepts HTTP status 300 in pagination. Result: It is possible to dynamically provision Cinder persistent volumes.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:10:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianwei Hou 2017-09-12 08:20:14 UTC
Description of problem:
Dynamically provision a Cinder PV, the Cinder volume can not be created. Error "Error creating cinder volume: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead"

Version-Release number of selected component (if applicable):
openshift v3.7.0-0.125.0
kubernetes v1.7.0+695f48a16f
etcd 3.2.1

How reproducible:
Always

Steps to Reproduce:
1. Setup OCP with standalone api and controller services.
2. Create a default storageclass
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
    creationTimestamp: 2017-09-12T07:27:11Z
    name: generic
    namespace: ""
    resourceVersion: "24496"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/generic
    uid: caf75660-978b-11e7-8799-fa163e765c12
  provisioner: kubernetes.io/cinder
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

3. Create a PVC
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      openshift.io/generated-by: OpenShiftNewApp
      volume.alpha.kubernetes.io/storage-class: foo
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
    creationTimestamp: 2017-09-12T07:30:01Z
    labels:
      app: mysql-persistent
      template: mysql-persistent-template
    name: mysql
    namespace: wzheng4
    resourceVersion: "24800"
    selfLink: /api/v1/namespaces/wzheng4/persistentvolumeclaims/mysql
    uid: 2feb4297-978c-11e7-8799-fa163e765c12
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: generic
  status:
    phase: Pending
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Actual results:
PV wasn't provisioned.

Name:		mysql
Namespace:	wzheng4
StorageClass:	generic
Status:		Pending
Volume:		
Labels:		app=mysql-persistent
		template=mysql-persistent-template
Annotations:	openshift.io/generated-by=OpenShiftNewApp
		volume.alpha.kubernetes.io/storage-class=foo
		volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder
Capacity:	
Access Modes:	
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----				-------------	--------	------			-------
  21m		1m		83	persistentvolume-controller			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "generic": Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead
{"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]}


Expected results:
PVs are provisioned successfully.


Master Log:
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]}
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: E0912 03:49:46.734186   18334 openstack.go:615] Error when retrieving list of supported blockstorage api versions: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]}
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: E0912 03:49:46.734199   18334 openstack_volumes.go:305] Unable to initialize cinder client for region: regionOne
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: I0912 03:49:46.734203   18334 cinder_util.go:207] Error creating cinder volume: Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: {"versions": [{"status": "DEPRECATED", "updated": "2016-05-02T20:25:19Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v1/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v1.0"}, {"status": "SUPPORTED", "updated": "2014-06-28T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v2/", "rel": "self"}], "min_version": "", "version": "", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v2.0"}, {"status": "CURRENT", "updated": "2016-02-08T12:20:21Z", "links": [{"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}, {"href": "https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/v3/", "rel": "self"}], "min_version": "3.0", "version": "3.15", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.volume+json;version=1"}], "id": "v3.0"}]}
Sep 12 03:49:46 host-8-241-63 atomic-openshift-master-controllers: I0912 03:49:46.734219   18334 pv_controller.go:1331] failed to provision volume for claim "wzheng4/mysql" with StorageClass "generic": Expected HTTP response code [200 204] when accessing [GET https://ci-rhos.centralci.eng.rdu2.redhat.com:13776/], but got 300 instead
...
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.460737   73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.460846   73893 wrap.go:42] GET /api/v1/nodes: (20.271503ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798]
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461142   73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461223   73893 wrap.go:42] GET /api/v1/nodes: (20.340152ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798]
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461329   73893 rbac.go:116] RBAC DENY: user "system:serviceaccount:kube-system:persistent-volume-binder" groups ["system:serviceaccounts" "system:serviceaccounts:kube-system" "system:authenticated"] cannot "list" resource "nodes" cluster-wide
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-api: I0912 03:53:34.461405   73893 wrap.go:42] GET /api/v1/nodes: (20.276866ms) 403 [[openshift/v1.7.0+695f48a16f (linux/amd64) kubernetes/8070990/system:serviceaccount:kube-system:persistent-volume-binder] 10.8.241.63:41798]
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462865   18334 cinder_util.go:149] Error listing nodes
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462887   18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462903   18334 pv_controller.go:1331] failed to provision volume for claim "wzheng4/mysql" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462981   18334 cinder_util.go:149] Error listing nodes
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462988   18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.462998   18334 pv_controller.go:1331] failed to provision volume for claim "wzheng3/redis" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463058   18334 cinder_util.go:149] Error listing nodes
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463065   18334 cinder_util.go:195] error getting zone information: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster
Sep 12 03:53:34 host-8-241-63 atomic-openshift-master-controllers: I0912 03:53:34.463074   18334 pv_controller.go:1331] failed to provision volume for claim "wzheng2/redis" with StorageClass "generic": User "system:serviceaccount:kube-system:persistent-volume-binder" cannot list all nodes in the cluster


Additional info:
Attached master log

Comment 1 Pavel Pospisil 2017-09-12 11:15:43 UTC
I assume that the OpenShift cloud.conf file has Cinder version automatic detection configured. Such configuration looks like:
[BlockStorage]
bs-version = auto

Please, can you confirm this?

Comment 2 Pavel Pospisil 2017-09-12 11:25:50 UTC
There's a workaround for this bug: configure in cloud.conf a specific Cinder version, e.g. V2:
[BlockStorage]
bs-version = v2

IMO, the problem is that in case OpenShift is configured to auto-detect Cider version it uses Cinder V1 to query for available Cinder versions, however, as the Cinder V1 is deprecated in the used OpenStack cluster the autodetection fails.

Comment 3 Jianwei Hou 2017-09-13 02:46:09 UTC
@ppospisi Thank you, that fixed the problem!

I'll transfer its component to docs to have this configuration item documented.

Comment 4 Jianwei Hou 2017-09-13 02:48:46 UTC
Sorry move back to storage to decide if we want to fix it first.

Comment 5 Pavel Pospisil 2017-09-13 10:26:11 UTC
I looked into the issue again an found out that the issue is fixed in gophercloud library [1] and in Kubernetes [2] for K8s 1.8 release.

So upgrading the gophercloud library to commit [3] in OpenShift vendor directory should fix the problem.

[1] https://github.com/gophercloud/gophercloud/pull/383
[2] https://github.com/kubernetes/kubernetes/pull/47602
[3] https://github.com/gophercloud/gophercloud/commit/ed590d9afe113c6107cd60717b196155e6579e78

Comment 6 Pavel Pospisil 2017-09-13 12:51:29 UTC
I backported the gophercloud library fix [1] and created PR [2].

[1] https://github.com/gophercloud/gophercloud/pull/383
[2] https://github.com/openshift/origin/pull/16321

Comment 7 Pavel Pospisil 2017-09-19 10:05:30 UTC
The PR https://github.com/openshift/origin/pull/16321 was merged.

Comment 9 Jianwei Hou 2017-09-22 08:27:20 UTC
Verified this is fixed in 3.7.0-0.127. Cinder volume provisioned successfully without the additional config from comment 1.

Comment 13 errata-xmlrpc 2017-11-28 22:10:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Comment 14 August Simonelli 2018-03-30 06:33:18 UTC
I just hit this on a brand new OCP 3.7 plus OSP 12. Did this with advanced install so not sure where to update cloud.conf with bs-version=v2 as i installed with openshift-ansible pointing at an inventory file. Can I add bs-version=v2 to the inventory at install time? Does it go under a subsection? [BlockStorage] ?

Comment 15 hchen 2018-04-02 13:15:05 UTC
Please refer to this doc [1]. Add to cloud.conf with the following:

[BlockStorage]
bs-version = v2


1. https://docs.openshift.com/container-platform/3.6/install_config/configuring_openstack.html#configuring-openstack-variables

Comment 16 August Simonelli 2018-04-02 13:17:58 UTC
With an advanced install of OCP 3.7 there is no cloud.conf. So you need this:

https://github.com/openshift/openshift-ansible/pull/6611/files

Basically you need to set the ansible variable:

openshift_cloudprovider_openstack_blockstorage_version

but it won't work without a change to the jinja2 template as well

roles/openshift_cloud_provider/templates/openstack.conf.j2

See link.

Also the link you shared doesn't mention the value required, so could be confusing.