Bug 1523625 - service catalog deployment fails
Summary: service catalog deployment fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.7.z
Assignee: Jeff Peeler
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1526150 1539634 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-08 12:23 UTC by Jaspreet Kaur
Modified: 2018-09-13 15:08 UTC (History)
31 users (show)

Fixed In Version: openshift-ansible-3.7.24-1.git.0.18a2c6a.el7
Doc Type: Bug Fix
Doc Text:
The ansible installer previously was not updating the api service definition with newly generated certificate data. Also, the service catalog api server wasn't being restarted to pick up the new certs either. Using mismatched CAs causes x509 errors in the api server logs and has now been corrected.
Clone Of:
Environment:
Last Closed: 2018-04-05 09:33:10 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0636 None None None 2018-04-05 09:33:53 UTC
Red Hat Knowledge Base (Solution) 3337731 None None None 2018-03-16 23:01:40 UTC
Red Hat Knowledge Base (Solution) 3349011 None None None 2018-03-15 16:57:01 UTC

Description Jaspreet Kaur 2017-12-08 12:23:43 UTC
Description of problem: 
Cant deploy service catalog. 


TASK [ansible_service_broker : Create the Broker resource in the catalog] *******************************************************************************************************
fatal: [dcscapgomaster01.sgdc.se]: FAILED! => {"changed": false, "failed": true, "msg": {"cmd": "/usr/local/bin/oc create -f /tmp/brokerout-3Uh_A1 -n default", "results": {}, "returncode": 1, "stderr": "error: unable to recognize \"/tmp/brokerout-3Uh_A1\": no matches for servicecatalog.k8s.io/, Kind=ClusterServiceBroker\n", "stdout": ""}}


E1206 08:35:34.358065       1 memcache.go:159] couldn't get resource list for servicecatalog.k8s.io/v1beta1: an error on the server ("Error: 'x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"service-catalog-signer\")'\nTrying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'") has prevented the request from succeeding


I1206 08:45:28.625696       1 controller_manager.go:213] Using namespace kube-service-catalog for leader election lock
I1206 08:45:28.625712       1 leaderelection.go:174] attempting to acquire leader lease...
I1206 08:45:28.626430       1 healthz.go:74] Installing healthz checkers:"ping", "checkAPIAvailableResources"
E1206 08:45:28.648962       1 event.go:260] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"service-catalog-controller-manager", GenerateName:"", Namespace:"kube-service-catalog", SelfLink:"/api/v1/namespaces/kube-service-catalog/endpoints/service-catalog-controller-manager", UID:"05bd5da7-da5c-11e7-ae4f-005056ba6d3a", ResourceVersion:"4627480", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:63648144242, nsec:0, loc:(*time.Location)(0x25df400)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"controller-manager-nvnms-external-service-catalog-controller\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2017-12-06T08:09:04Z\",\"renewTime\":\"2017-12-06T08:45:28Z\",\"leaderTransitions\":1}"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'no kind is registered for the type v1.Endpoints'. Will not report event: 'Normal' 'LeaderElection' 'controller-manager-nvnms-external-service-catalog-controller became leader'
I1206 08:45:28.649054       1 leaderelection.go:184] successfully acquired lease kube-service-catalog/service-catalog-controller-manager
I1206 08:45:28.649103       1 controller_manager.go:297] Getting available resources
I1206 08:45:28.649378       1 controller_manager.go:259] Created client for API discovery
I1206 08:45:28.683898       1 request.go:1038] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "service-catalog-signer")'
Trying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'' 
I1206 08:45:28.711786       1 request.go:1038] body was not decodable (unable to check for Status): Object 'Kind' is missing in 'Error: 'x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "service-catalog-signer")'
Trying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1'' 
F1206 08:45:28.713230       1 controller_manager.go:198] error running controllers: failed to get supported resources from server: unable to retrieve the complete list of server APIs: servicecatalog.k8s.io/v1beta1: an error on the server ("Error: 'x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"service-catalog-signer\")'\nTrying to reach: 'https://172.30.72.245:443/apis/servicecatalog.k8s.io/v1beta1' ") has prevented the request from succeeding

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results: Fails everytime

Expected results: should deploy successfully.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 6 Jeff Peeler 2018-01-10 20:43:00 UTC
*** Bug 1526150 has been marked as a duplicate of this bug. ***

Comment 7 Jeff Peeler 2018-01-10 20:47:39 UTC
Upstream PR: https://github.com/openshift/openshift-ansible/pull/6687

Comment 37 Scott Dodson 2018-02-05 18:19:10 UTC
The workaround for this bug is to update the service catalog apiservice. Set the ca_bundle field to the base64 encoded contents of /etc/origin/service-catalog/ca.crt and then delete the apiservice pod. When it's recreated it should work.

cat /etc/origin/service-catalog/ca.crt | base64

oc edit apiservice/v1beta1.servicecatalog.k8s.io

update ca_bundle field with the base64 encoded content from the first command

Comment 38 John Matthews 2018-02-26 16:07:42 UTC
*** Bug 1539634 has been marked as a duplicate of this bug. ***

Comment 48 errata-xmlrpc 2018-04-05 09:33:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0636


Note You need to log in before you can comment on or make changes to this bug.