Bug 1471062 - [RFE] etcd pod is in pending status due to pv is not created when deploying service catalog onto an existing cluster
Summary: [RFE] etcd pod is in pending status due to pv is not created when deploying s...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.7.0
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1467574
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-14 10:35 UTC by Johnny Liu
Modified: 2019-02-18 18:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-18 18:03:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Johnny Liu 2017-07-14 10:35:14 UTC
Description of problem:
On an existing cluster, enable service catalog deployment, at final, in openshift-ansible-service-broker project, asb and etcd pod are not running. due to no pv is created for etcd pvc.

Version-Release number of the following components:
openshift-ansible-3.6.144-1.git.0.50e12bf.el7.noarch
ansible-2.2.3.0-1.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. deploy a normal cluster
2. enable service catalog in inventory host file
openshift_hosted_etcd_storage_kind=nfs
openshift_hosted_etcd_storage_nfs_directory=/opt/osev3-etcd
openshift_hosted_etcd_storage_volume_name=etcd-vol3
openshift_hosted_etcd_storage_access_modes=['ReadWriteOnce']
openshift_hosted_etcd_storage_volume_size=10G
openshift_hosted_etcd_storage_host=10.8.183.187
openshift_hosted_etcd_storage_labels="{'storage': 'etcd'}"

openshift_enable_service_catalog=true
openshift_service_catalog_image_version=latest
openshift_service_catalog_image_prefix=openshift3/ose-
ansible_service_broker_image_prefix=openshift3/
ansible_service_broker_registry_url="registry.access.stage.redhat.com"

3. deploy service catalog
# ansible-playbook -i inventory_file ./playbooks/byo/openshift-cluster/service-catalog.yml

4. playbook is finished successfully, go to openshift-ansible-service-broker project to check pod status.


Actual results:
# oc get po
NAME                    READY     STATUS              RESTARTS   AGE
asb-281511082-q8s97     0/1       CrashLoopBackOff    14         54m
etcd-1487946270-46j9s   0/1       pending   

# oc describe po etcd-1487946270-46j9s
<--snip-->
Events:
  FirstSeen	LastSeen	Count	From						SubObjectPath		Type		Reason			Message
  ---------	--------	-----	----						-------------		--------	------			-------
  1h		1h		53	default-scheduler							Warning		FailedScheduling	SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "etcd", which is unexpected. (repeated 2 times)
  1h		1h		1	default-scheduler							Warning		FailedScheduling	no nodes available to schedule pods
  1h		42m		124	default-scheduler							Warning		FailedScheduling	SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "etcd", which is unexpected. (repeated 2 times)


# oc get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
etcd      Pending                                                     46m

# oc get pv
NAME               CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE
regpv-volume       17G        RWX           Retain          Bound     default/regpv-claim                                              1h

No etcd pv is created.

Expected results:
etcd pv is created created, and all pods are running well.


Additional info:
manually create etcd pv, pods are running well.

Comment 2 Scott Dodson 2017-07-20 14:47:35 UTC

*** This bug has been marked as a duplicate of bug 1467574 ***

Comment 3 Johnny Liu 2017-07-20 15:13:19 UTC
I do not think this bug is a dup of bug 1467574, the root cause of this bug is the following:
when running the separate service catalog playbook on an existing cluster, openshift_persistent_volumes is never called, so no pv is created. If running playbooks/byo/config.yml, no such issue.

While in BZ#1467574, it is running playbooks/byo/config.yml, but did not set openshift_hosted_etcd_storage to an existing nfs dir, want installer help create the nfs dir automatically. It is tracking installer does not create new nfs export dir on NFS server for etcd pv, the root cause is openshift_storage_nfs never take openshift_hosted_etcd_storage into consideration.

Comment 4 Scott Dodson 2017-07-20 15:22:07 UTC
ok, depends on it then, we'll work on this post 3.6 GA

Comment 5 Scott Dodson 2017-08-14 15:45:44 UTC
Need to add the persistent volumes playbooks from hosted into the service catalog playbook. Not going to get this for 3.6.1 but will be a 3.7 blocker.

Comment 11 ewolinet 2017-10-09 20:50:46 UTC
Dropping priority/severity and marking it as a RFE (but keeping it under installer).

At this time a PV can be created and then the service_catalog.yml playbook can be run which would allow the etcd PVC to bind.

Comment 12 Scott Dodson 2019-02-18 18:03:09 UTC
There appear to be no active cases related to this bug. As such we're closing this bug in order to focus on bugs that are still tied to active customer cases. Please re-open this bug if you feel it was closed in error or a new active case is attached.

Also, this implementation is no longer relevant in 3.9 and later where etcd is no longer necessary for service catalog.


Note You need to log in before you can comment on or make changes to this bug.