Bug 1689366 - kube-service-catalog/service-catalog-controller-manager fails to start with API Logging level 8+
Summary: kube-service-catalog/service-catalog-controller-manager fails to start with A...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Master
Version: 3.11.0
Hardware: Unspecified
OS: All
Target Milestone: ---
: 3.11.z
Assignee: Stefan Schimanski
QA Contact: Xingxing Xia
: 1663501 1695461 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2019-03-15 17:51 UTC by Mangirdas Judeikis
Modified: 2019-07-10 16:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-04-11 05:38:40 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1680342 0 unspecified CLOSED Installer failed at [openshift_service_catalog : Wait for Controller Manager rollout success] 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2019:0636 0 None None None 2019-04-11 05:38:45 UTC

Description Mangirdas Judeikis 2019-03-15 17:51:19 UTC
Description of problem:

Service-Catalog-controller fails to start if api server logging is set to 8.


Set API and controller log level to 8. 
Leave SC logging level at 3. 
Initiate cluster creation. 

service-catalog-controller fails to start on callback error:
[root@master-000000 ~]# oc get pods
NAME                       READY     STATUS             RESTARTS   AGE
apiserver-ft9zs            1/1       Running            0          4m
apiserver-v7wwl            1/1       Running            0          4m
apiserver-z8wgq            1/1       Running            0          4m
controller-manager-g9rcr   0/1       CrashLoopBackOff   4          4m
controller-manager-kqnrj   0/1       CrashLoopBackOff   4          4m
controller-manager-z5tn2   1/1       Running            4          4m
[root@master-000000 ~]# oc logs -f controller-manager-g9rcr
I0315 17:44:40.203181       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true]
I0315 17:44:40.203455       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true]
I0315 17:44:40.203496       1 feature_gate.go:194] feature gates: map[OriginatingIdentity:true AsyncBindingOperations:true NamespacedServiceBroker:true]
I0315 17:44:40.203540       1 hyperkube.go:192] Service Catalog version v3.11.82;Upstream:v0.1.35 (built 2019-02-08T01:46:38Z)
I0315 17:44:40.208482       1 leaderelection.go:185] attempting to acquire leader lease  kube-service-catalog/service-catalog-controller-manager...
I0315 17:44:57.676956       1 leaderelection.go:194] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I0315 17:44:57.677397       1 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-service-catalog", Name:"service-catalog-controller-manager", UID:"9537297a-4749-11e9-8cec-000d3a45ae44", APIVersion:"v1", ResourceVersion:"5588", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' controller-manager-g9rcr-external-service-catalog-controller became leader
F0315 17:44:57.786691       1 controller_manager.go:237] error running controllers: failed to get api versions from server: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: an error on the server ("unable to set dialer for kube-service-catalog/apiserver as rest transport is of type *transport.debuggingRoundTripper") has prevented the request from succeeding


Version-Release number of selected component (if applicable):

[root@master-000000 ~]# oc version
oc v3.11.82
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dnjasyruadbrlawjcpsx.westeurope.cloudapp.azure.com
openshift v3.11.82
kubernetes v1.11.0+d4cacc0

Comment 1 Mangirdas Judeikis 2019-03-15 18:24:19 UTC
API loglevel: 2
Controller: 8, works just fine.

Comment 4 Xingxing Xia 2019-03-21 09:23:53 UTC
Verified in 3.11.98, after API loglevel set to 8, restart SC controller-manager, it can be Running:
oc rsh -n kube-system master-api-ip-172-18-15-137.ec2.internal
sh-4.2# ps -eF | grep root
root          1      0 66 319117 671756 1 05:11 ?        00:00:45 openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8

oc delete pod --all -n kube-service-catalog

oc get pod -n kube-service-catalog -w
NAME                       READY     STATUS    RESTARTS   AGE
apiserver-9766n            1/1       Running   0          5m
controller-manager-frf2m   1/1       Running   0          5m

Comment 6 errata-xmlrpc 2019-04-11 05:38:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 7 Jesus M. Rodriguez 2019-07-10 16:01:17 UTC
*** Bug 1695461 has been marked as a duplicate of this bug. ***

Comment 8 Jesus M. Rodriguez 2019-07-10 16:02:41 UTC
*** Bug 1663501 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.