Bug 1491331

Summary:	failed to provision volume for claim test/pv0001-claim with StorageClass cinder. No suitable endpoint could be found in the service catalog
Product:	OpenShift Container Platform	Reporter:	Javier Ramirez <javier.ramirez>
Component:	Storage	Assignee:	Pavel Pospisil <ppospisi>
Status:	CLOSED ERRATA	QA Contact:	Qin Ping <piqin>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	3.6.1	CC:	aos-bugs, aos-storage-staff, bchilds, bward, hchen, jack.ottofaro, javier.ramirez, jliberma, misalunk, ppospisi, wsun
Target Milestone:	---	Keywords:	Reopened
Target Release:	3.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Dynamic provisioning of Cinder volumes stops working after certain amount of time. OpenShift restart resolves this problem temporarily. That's because OpenShift is not able to re-authenticate to OpenStack Keystone V3. Consequence: It is not possible to create new Cinder volumes using dynamic provisioning. Fix: The OpenShift re-authentication to OpenStack Keystone V3 was fixed. Result: Dynamic provisioning of new Cinder volumes works as documented.	Story Points:	---
Clone Of:
Clones:	1533360 1563085 (view as bug list)		Environment:
Last Closed:	2017-11-28 22:10:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1533360, 1563085

Description Javier Ramirez 2017-09-13 14:10:47 UTC

Description of problem:
Creating a new pvc using cinder storageclass results in the pvc being stuck in a pending state. restart openshift services on masters ( systemctl restart atomic-openshift-*) resolves the issue temporarily and the volume is created in the cinder backend. After some time the pvc is stuck pending again until we restart services. 


Version-Release number of selected component (if applicable):
contianerised install of OCP 3.6 (v3.6.173.0.5-5)

How reproducible:
Always

Steps to Reproduce:
1. Creating a new pvc using cinder storageclass
2.
3.

Actual results:
pvc being stuck in a pending state. 

Expected results:
Volume to be created in the cinder backend.

Error:
"failed to provision volume for claim test/pv0001-claim with StorageClass cinder. No suitable endpoint could be found in the service catalog."

Additional info:
Seems to be similar to: https://github.com/openshift/origin/issues/15979

Workaround seems to be re-configured the cloudprovider openstack.conf to use keystone v2.0 with a local OpenStack user.

Same workaround was mentioned in another bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1490768 - Bug 1490768 - Failed to provision cinder volume, got http response code 300

Comment 2 Pavel Pospisil 2017-09-13 14:15:39 UTC

Please, will you provide cloud.conf configuration file.

Will you, please, also provide OpenStack service catalog, especially the Cinder entries.

Comment 22 Qin Ping 2017-11-08 03:30:44 UTC

Verified this bug in OCP3.7 on our openstack

openshift v3.7.0-0.189.0
kubernetes v1.7.6+a08f5eeb62

After reauth, PVC can be created successfully and get into "Bound" phase.

So marked this bug as verified.

Comment 26 errata-xmlrpc 2017-11-28 22:10:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188

Comment 30 Brian Ward 2018-06-28 13:06:50 UTC

Note: Backport to 3.6 provided in 1527973.