Bug 1689366

Summary: kube-service-catalog/service-catalog-controller-manager fails to start with API Logging level 8+
Product: OpenShift Container Platform Reporter: Mangirdas Judeikis <mjudeiki>
Component: MasterAssignee: Stefan Schimanski <sttts>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.11.0CC: aos-bugs, bmchugh, jokerman, mfojtik, mmccomas, pkanthal
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-11 05:38:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mangirdas Judeikis 2019-03-15 17:51:19 UTC
Description of problem:

Service-Catalog-controller fails to start if api server logging is set to 8.

Reproduce:

Set API and controller log level to 8. 
Leave SC logging level at 3. 
Initiate cluster creation. 

service-catalog-controller fails to start on callback error:
```
[root@master-000000 ~]# oc get pods
NAME                       READY     STATUS             RESTARTS   AGE
apiserver-ft9zs            1/1       Running            0          4m
apiserver-v7wwl            1/1       Running            0          4m
apiserver-z8wgq            1/1       Running            0          4m
controller-manager-g9rcr   0/1       CrashLoopBackOff   4          4m
controller-manager-kqnrj   0/1       CrashLoopBackOff   4          4m
controller-manager-z5tn2   1/1       Running            4          4m
[root@master-000000 ~]# oc logs -f controller-manager-g9rcr
I0315 17:44:40.203181       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I0315 17:44:40.203455       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true]
I0315 17:44:40.203496       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true NamespacedServiceBroker:true]
I0315 17:44:40.203540       1 hyperkube.go:192] Service Catalog version v3.11.82;Upstream:v0.1.35 (built 2019-02-08T01:46:38Z)
I0315 17:44:40.208482       1 leaderelection.go:185] attempting to acquire leader lease  kube-service-catalog/service-catalog-controller-manager...
I0315 17:44:57.676956       1 leaderelection.go:194] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I0315 17:44:57.677397       1 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-service-catalog", Name:"service-catalog-controller-manager", UID:"9537297a-4749-11e9-8cec-000d3a45ae44", APIVersion:"v1", ResourceVersion:"5588", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' controller-manager-g9rcr-external-service-catalog-controller became leader
F0315 17:44:57.786691       1 controller_manager.go:237] error running controllers: failed to get api versions from server: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: an error on the server ("unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper") has prevented the request from succeeding

```


Version-Release number of selected component (if applicable):

[root@master-000000 ~]# oc version
oc v3.11.82
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dnjasyruadbrlawjcpsx.westeurope.cloudapp.azure.com
openshift v3.11.82
kubernetes v1.11.0+d4cacc0

Comment 1 Mangirdas Judeikis 2019-03-15 18:24:19 UTC
API loglevel: 2
Controller: 8, works just fine.

Comment 4 Xingxing Xia 2019-03-21 09:23:53 UTC
Verified in 3.11.98, after API loglevel set to 8, restart SC controller-manager, it can be Running:
oc rsh -n kube-system master-api-ip-172-18-15-137.ec2.internal
sh-4.2# ps -eF | grep root
root          1      0 66 319117 671756 1 05:11 ?        00:00:45 openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8

oc delete pod --all -n kube-service-catalog

oc get pod -n kube-service-catalog -w
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-9766n            1/1       Running   0          5m
controller-manager-frf2m   1/1       Running   0          5m

Comment 6 errata-xmlrpc 2019-04-11 05:38:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636

Comment 7 Jesus M. Rodriguez 2019-07-10 16:01:17 UTC
*** Bug 1695461 has been marked as a duplicate of this bug. ***

Comment 8 Jesus M. Rodriguez 2019-07-10 16:02:41 UTC
*** Bug 1663501 has been marked as a duplicate of this bug. ***