Bug 1564828

Summary:	Service catalog controller pod in crash state after install
Product:	OpenShift Container Platform	Reporter:	Weihua Meng <wmeng>
Component:	Installer	Assignee:	Jay Boyd <jaboyd>
Status:	CLOSED ERRATA	QA Contact:	Weihua Meng <wmeng>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, chezhang, jaboyd, jmatthew, jokerman, mmccomas, wzheng, xiuwang, xtian
Target Milestone:	---
Target Release:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	bug was never in a customer release	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-07-30 19:12:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Weihua Meng 2018-04-08 04:23:59 UTC

Description of problem:
Service catalog controller pod in crash state after install

Version-Release number of the following components:
openshift-ansible-3.10.0-0.15.0.git.0.556ddbb.el7.noarch.rpm
openshift v3.10.0-0.16.0

How reproducible:
Always

Steps to Reproduce:
1. install OCP 3.10
2. check cluster status

Actual results:
# oc get pods --all-namespaces
NAMESPACE                           NAME                             READY     STATUS             RESTARTS   AGE
default                             docker-registry-1-6sxrx          1/1       Running            0          28m
default                             docker-registry-1-7czkz          1/1       Running            0          28m
default                             registry-console-1-gb6r7         1/1       Running            0          26m
default                             router-1-2s74x                   1/1       Running            0          28m
default                             router-1-9mcbl                   1/1       Running            0          28m
install-test                        mongodb-1-d5mth                  1/1       Running            0          24m
install-test                        nodejs-mongodb-example-1-build   0/1       Completed          0          24m
install-test                        nodejs-mongodb-example-1-qg76f   1/1       Running            0          23m
kube-service-catalog                apiserver-k52wg                  1/1       Running            0          25m
kube-service-catalog                controller-manager-44zb5         0/1       CrashLoopBackOff   9          25m
openshift-ansible-service-broker    asb-1-rzzgj                      1/1       Running            2          25m
openshift-ansible-service-broker    asb-etcd-1-8gdd6                 1/1       Running            0          25m
openshift-node                      sync-5zq86                       1/1       Running            0          31m
openshift-node                      sync-7bw92                       1/1       Running            0          31m
openshift-node                      sync-nthwj                       1/1       Running            0          31m
openshift-sdn                       ovs-6h8qh                        1/1       Running            0          31m
openshift-sdn                       ovs-jsgsp                        1/1       Running            0          31m
openshift-sdn                       ovs-nb2jc                        1/1       Running            0          31m
openshift-sdn                       sdn-44b65                        1/1       Running            0          31m
openshift-sdn                       sdn-74zsj                        1/1       Running            0          31m
openshift-sdn                       sdn-j98jb                        1/1       Running            0          31m
openshift-template-service-broker   apiserver-q5prx                  1/1       Running            0          24m
openshift-web-console               webconsole-86df987b49-sldx7      1/1       Running            0          27m

# oc logs controller-manager-44zb5 -n kube-service-catalog
Flag --port has been deprecated, see --secure-port instead
I0408 03:05:47.219369       1 feature_gate.go:184] feature gates: map[OriginatingIdentity:true]
I0408 03:05:47.219573       1 hyperkube.go:188] Service Catalog version v0.0.0-master+$Format:%h$ (built 2018-04-04T13:43:16Z)
W0408 03:05:47.219623       1 controller_manager.go:97] program option --port is obsolete and ignored, specify --secure-port instead
I0408 03:05:47.219645       1 controller_manager.go:101] Building k8s kubeconfig
I0408 03:05:47.223408       1 controller_manager.go:129] Building service-catalog kubeconfig for url: 
I0408 03:05:47.223429       1 controller_manager.go:136] Using inClusterConfig to talk to service catalog API server -- make sure your API server is registered with the aggregator
Error: failed to establish SecureServingOptions error reading /var/run/kubernetes-service-catalog/apiserver.key, certificate and key must be supplied as a pair

image used: openshift3/ose-service-catalog:v3.10.0-0.16.0.0

Expected results:
No crash pods after install

Comment 1 Zhang Cheng 2018-04-10 05:29:54 UTC

Add "TestBlocker" keyword since it is blocking all TCs of service-catalog & asb in OCP3.10

Comment 2 Jay Boyd 2018-04-10 16:06:42 UTC

I believe this is fixed by https://github.com/openshift/openshift-ansible/pull/7681

I'm attempting to verify.

Comment 3 Jay Boyd 2018-04-10 18:05:32 UTC

This is fixed in Origin, it was merged on April 7 with https://github.com/openshift/openshift-ansible/pull/7681

Comment 4 Xiaoli Tian 2018-04-11 05:09:24 UTC

available to test on openshift-ansible-3.10.0-0.19.0

Comment 5 Weihua Meng 2018-04-12 09:49:21 UTC

Test blocked by bz1565525

Comment 6 Weihua Meng 2018-04-12 14:09:12 UTC

Fixed.
openshift-ansible-3.10.0-0.20.0.git.0.37bab0f.el7.noarch.rpm
# openshift version
openshift v3.10.0-0.20.0
kubernetes v1.10.0+b81c8f8
etcd 3.2.16

# oc get pods -n kube-service-catalog
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-2pgxm            1/1       Running   0          5h
controller-manager-d4tds   1/1       Running   0          5h

Comment 8 errata-xmlrpc 2018-07-30 19:12:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816