Bug 1273739

Summary:	Event shows "Cloud provider not initialized properly" when creating pod with cinder PV
Product:	OpenShift Container Platform	Reporter:	Jianwei Hou <jhou>
Component:	Storage	Assignee:	Mark Turansky <mturansk>
Status:	CLOSED ERRATA	QA Contact:	Liang Xia <lxia>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.0.0	CC:	aos-bugs, bchilds, bleanhar, jsafrane, pep, pruan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-01-26 19:16:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1267746

Description Jianwei Hou 2015-10-21 07:18:57 UTC

Description of problem:
Create a PV and PVC for cinder volume, the create a pod to mount the PV, pod can not be created successfully because Cloud provider not initialized properly.

In kubernetes, we start kube-apiserver and kubelet with flags '--cloud-provider' and '--cloud-config' to make openstack cloud provider initialized properly. In openshift, I think we can do the same thing to have cinder supported as part of the installation work, or have a documentation for manual installation

Version-Release number of selected component (if applicable):
openshift v3.0.2.901-61-g568adb6
kubernetes v1.1.0-alpha.1-653-g86b4e77

How reproducible:
Always

Steps to Reproduce:
1.  Create PV and PVC
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/cinder/pv-rwo-recycle.json
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/cinder/pvc-rwo.json

2. Create pod
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/cinder/pod.json

3.oc get pods; oc get events

Actual results:
Events indicated a failed mount:

9s          9s         1         cinderpd      Pod                                                         Scheduled          {scheduler }                                     Successfully assigned cinderpd to openshift-140.lab.eng.nay.redhat.com
10s         0s         2         cinderpd      Pod                                                         FailedSync         {kubelet openshift-140.lab.eng.nay.redhat.com}   Error syncing pod, skipping: Cloud provider not initialized properly
10s         0s         2         cinderpd      Pod                                                         FailedMount        {kubelet openshift-140.lab.eng.nay.redhat.com}   Unable to mount volumes for pod "cinderpd_jhou": Cloud provider not initialized properly


Expected results:
Should be able to create the pod successfully

Additional info:

Comment 2 Jianwei Hou 2015-10-21 07:50:21 UTC

Sorry, I wasn't aware that the doc is here https://github.com/jsafrane/openshift-docs/blob/195df5f3a67e267d7b67d0511e8e72abc50cd624/admin_guide/configuring_openstack.adoc

I followed the doc to restart node services, and when I run 'service atomic-openshift-node status -l', I got the error:

Oct 21 15:44:27 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: I1021 15:44:27.039291    9392 openstack.go:202] Claiming to support Instances
Oct 21 15:44:27 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: E1021 15:44:27.089563    9392 kubelet.go:845] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
Oct 21 15:44:34 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: I1021 15:44:34.105614    9392 openstack.go:202] Claiming to support Instances
Oct 21 15:44:34 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: E1021 15:44:34.153838    9392 kubelet.go:845] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
Oct 21 15:44:41 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: I1021 15:44:41.170865    9392 openstack.go:202] Claiming to support Instances
Oct 21 15:44:41 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: E1021 15:44:41.217828    9392 kubelet.go:845] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
Oct 21 15:44:48 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: I1021 15:44:48.237796    9392 openstack.go:202] Claiming to support Instances
Oct 21 15:44:48 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: E1021 15:44:48.344387    9392 kubelet.go:845] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object
Oct 21 15:44:55 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: I1021 15:44:55.362615    9392 openstack.go:202] Claiming to support Instances
Oct 21 15:44:55 openshift-140.lab.eng.nay.redhat.com atomic-openshift-node[9392]: E1021 15:44:55.413861    9392 kubelet.go:845] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: Failed to find object


I seems this issue https://github.com/kubernetes/kubernetes/issues/13556 has not been fixed. We will need work around for our ose setup to test cinder.

Comment 3 Jan Safranek 2015-10-21 08:35:16 UTC

I have a patch in the pipeline, which stro[s domain names from hostname: https://github.com/kubernetes/kubernetes/pull/15537

I'll try to push it to Origin - tracked in https://github.com/openshift/origin/pull/5272

Comment 4 Jianwei Hou 2015-11-11 12:56:01 UTC

Verified on 
openshift v3.1.0.4
kubernetes v1.1.0-origin-1107-g4c8e6f4
etcd 2.1.

This bug is fixed now, cinder pods can be created successfully!

Comment 6 errata-xmlrpc 2016-01-26 19:16:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0070