Bug 1523625

Summary: service catalog deployment fails
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: Service BrokerAssignee: Jeff Peeler <jpeeler>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.7.0CC: abutcher, aivaras.laimikis, aos-bugs, bmchugh, chezhang, chrkim, dcaldwel, dmoessne, erjones, fshaikh, ggore, jiazha, jkaur, jmalde, jokerman, jpeeler, jrosenta, mmccomas, mrobson, nbhatt, nnosenzo, pmorie, rbost, rhowe, sgaikwad, smunilla, snalawad, tibrahim, vwalek, wdecoste, wmeng
Target Milestone: ---   
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.7.24-1.git.0.18a2c6a.el7 Doc Type: Bug Fix
Doc Text:
The ansible installer previously was not updating the api service definition with newly generated certificate data. Also, the service catalog api server wasn't being restarted to pick up the new certs either. Using mismatched CAs causes x509 errors in the api server logs and has now been corrected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-05 09:33:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jaspreet Kaur 2017-12-08 12:23:43 UTC
Description of problem: 
Cant deploy service catalog. 


TASK [ansible_service_broker : Create the Broker resource in the catalog] *******************************************************************************************************
fatal: [dcscapgomaster01.sgdc.se]: FAILED! => {"changed": false, "failed": true, "msg": {"cmd": "/usr/local/bin/oc create -f /tmp/brokerout-3Uh_A1 -n default", "results": {}, "returncode": 1, "stderr": "error: unable to recognize \"/tmp/brokerout-3Uh_A1\": no matches for servicecatalog.k8s.io/, Kind=ClusterServiceBroker\n", "stdout": ""}}


E1206 08:35:34.358065       1 memcache.go:159] couldn't get resource list for servicecatalog.k8s.io/v1beta1: an error on the server ("Error: 'x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"service-catalog-signer\")'\nTrying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'") has prevented the request from succeeding


I1206 08:45:28.625696       1 controller_manager.go:213] Using namespace kube-service-catalog for leader election lock
I1206 08:45:28.625712       1 leaderelection.go:174] attempting to acquire leader lease...
I1206 08:45:28.626430       1 healthz.go:74] Installing healthz checkers:"ping", "checkAPIAvailableResources"
E1206 08:45:28.648962       1 event.go:260] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"service-catalog-controller-manager", GenerateName:"", Namespace:"kube-service-catalog", SelfLink:"/api/v1/namespaces/kube-service-catalog/endpoints/service-catalog-controller-manager", UID:"05bd5da7-da5c-11e7-ae4f-005056ba6d3a", ResourceVersion:"4627480", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:63648144242, nsec:0, loc:(*time.Location)(0x25df400)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"controller-manager-nvnms-external-service-catalog-controller\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-12-06T08:09:04Z\",\"renewTime\":\"2017-12-06T08:45:28Z\",\"leaderTransitions\":1}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'no kind is registered for the type v1.Endpoints'. Will not report event: 'Normal' 'LeaderElection' 'controller-manager-nvnms-external-service-catalog-controller became leader'
I1206 08:45:28.649054       1 leaderelection.go:184] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I1206 08:45:28.649103       1 controller_manager.go:297] Getting available resources
I1206 08:45:28.649378       1 controller_manager.go:259] Created client for API discovery
I1206 08:45:28.683898       1 request.go:1038] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "service-catalog-signer")'
Trying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'' 
I1206 08:45:28.711786       1 request.go:1038] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "service-catalog-signer")'
Trying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'' 
F1206 08:45:28.713230       1 controller_manager.go:198] error running controllers: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: an error on the server ("Error: 'x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"service-catalog-signer\")'\nTrying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1' ") has prevented the request from succeeding

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results: Fails everytime

Expected results: should deploy successfully.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 6 Jeff Peeler 2018-01-10 20:43:00 UTC
*** Bug 1526150 has been marked as a duplicate of this bug. ***

Comment 7 Jeff Peeler 2018-01-10 20:47:39 UTC
Upstream PR: https://github.com/openshift/openshift-ansible/pull/6687

Comment 37 Scott Dodson 2018-02-05 18:19:10 UTC
The workaround for this bug is to update the service catalog apiservice. Set the ca_bundle field to the base64 encoded contents of /etc/origin/service-catalog/ca.crt and then delete the apiservice pod. When it's recreated it should work.

cat /etc/origin/service-catalog/ca.crt | base64

oc edit apiservice/v1beta1.servicecatalog.k8s.io

update ca_bundle field with the base64 encoded content from the first command

Comment 38 John Matthews 2018-02-26 16:07:42 UTC
*** Bug 1539634 has been marked as a duplicate of this bug. ***

Comment 48 errata-xmlrpc 2018-04-05 09:33:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636