Bug 1471062

Summary: [RFE] etcd pod is in pending status due to pv is not created when deploying service catalog onto an existing cluster
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED WONTFIX QA Contact: Johnny Liu <jialiu>
Severity: low Docs Contact:
Priority: low    
Version: 3.6.0CC: aos-bugs, jialiu, jmatthew, jokerman, mmccomas
Target Milestone: ---Keywords: Reopened
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-18 18:03:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1467574    
Bug Blocks:    

Description Johnny Liu 2017-07-14 10:35:14 UTC
Description of problem:
On an existing cluster, enable service catalog deployment, at final, in openshift-ansible-service-broker project, asb and etcd pod are not running. due to no pv is created for etcd pvc.

Version-Release number of the following components:
openshift-ansible-3.6.144-1.git.0.50e12bf.el7.noarch
ansible-2.2.3.0-1.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. deploy a normal cluster
2. enable service catalog in inventory host file
openshift_hosted_etcd_storage_kind=nfs
openshift_hosted_etcd_storage_nfs_directory=/opt/osev3-etcd
openshift_hosted_etcd_storage_volume_name=etcd-vol3
openshift_hosted_etcd_storage_access_modes=['ReadWriteOnce']
openshift_hosted_etcd_storage_volume_size=10G
openshift_hosted_etcd_storage_host=10.8.183.187
openshift_hosted_etcd_storage_labels="{'storage': 'etcd'}"

openshift_enable_service_catalog=true
openshift_service_catalog_image_version=latest
openshift_service_catalog_image_prefix=openshift3/ose-
ansible_service_broker_image_prefix=openshift3/
ansible_service_broker_registry_url="registry.access.stage.redhat.com"

3. deploy service catalog
# ansible-playbook -i inventory_file ./playbooks/byo/openshift-cluster/service-catalog.yml

4. playbook is finished successfully, go to openshift-ansible-service-broker project to check pod status.


Actual results:
# oc get po
NAME                    READY     STATUS              RESTARTS   AGE
asb-281511082-q8s97     0/1       CrashLoopBackOff    14         54m
etcd-1487946270-46j9s   0/1       pending   

# oc describe po etcd-1487946270-46j9s
<--snip-->
Events:
  FirstSeen	LastSeen	Count	From						SubObjectPath		Type		Reason			Message
  ---------	--------	-----	----						-------------		--------	------			-------
  1h		1h		53	default-scheduler							Warning		FailedScheduling	SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "etcd", which is unexpected. (repeated 2 times)
  1h		1h		1	default-scheduler							Warning		FailedScheduling	no nodes available to schedule pods
  1h		42m		124	default-scheduler							Warning		FailedScheduling	SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "etcd", which is unexpected. (repeated 2 times)


# oc get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
etcd      Pending                                                     46m

# oc get pv
NAME               CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE
regpv-volume       17G        RWX           Retain          Bound     default/regpv-claim                                              1h

No etcd pv is created.

Expected results:
etcd pv is created created, and all pods are running well.


Additional info:
manually create etcd pv, pods are running well.

Comment 2 Scott Dodson 2017-07-20 14:47:35 UTC

*** This bug has been marked as a duplicate of bug 1467574 ***

Comment 3 Johnny Liu 2017-07-20 15:13:19 UTC
I do not think this bug is a dup of bug 1467574, the root cause of this bug is the following:
when running the separate service catalog playbook on an existing cluster, openshift_persistent_volumes is never called, so no pv is created. If running playbooks/byo/config.yml, no such issue.

While in BZ#1467574, it is running playbooks/byo/config.yml, but did not set openshift_hosted_etcd_storage to an existing nfs dir, want installer help create the nfs dir automatically. It is tracking installer does not create new nfs export dir on NFS server for etcd pv, the root cause is openshift_storage_nfs never take openshift_hosted_etcd_storage into consideration.

Comment 4 Scott Dodson 2017-07-20 15:22:07 UTC
ok, depends on it then, we'll work on this post 3.6 GA

Comment 5 Scott Dodson 2017-08-14 15:45:44 UTC
Need to add the persistent volumes playbooks from hosted into the service catalog playbook. Not going to get this for 3.6.1 but will be a 3.7 blocker.

Comment 11 ewolinet 2017-10-09 20:50:46 UTC
Dropping priority/severity and marking it as a RFE (but keeping it under installer).

At this time a PV can be created and then the service_catalog.yml playbook can be run which would allow the etcd PVC to bind.

Comment 12 Scott Dodson 2019-02-18 18:03:09 UTC
There appear to be no active cases related to this bug. As such we're closing this bug in order to focus on bugs that are still tied to active customer cases. Please re-open this bug if you feel it was closed in error or a new active case is attached.

Also, this implementation is no longer relevant in 3.9 and later where etcd is no longer necessary for service catalog.