Bug 1603343

Summary: "Error: 'x509: certificate signed by unknown authority" at "TASK [ansible_service_broker : Create the Broker resource in the catalog]"
Product: OpenShift Container Platform Reporter: François Cami <fcami>
Component: Service CatalogAssignee: Dan Geoffroy <dageoffr>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: high Docs Contact:
Priority: high    
Version: 3.9.0CC: aos-bugs, chezhang, cshereme, fabian, fcami, jiazha, jmatthew, jokerman, mmccomas, sdodson, vwalek, wmeng, zitang
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:40:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description François Cami 2018-07-19 14:29:11 UTC
Description of problem:

     Play:     Upgrade Service Catalog
     Task:     Create the Broker resource in the catalog
     Message:  {u'cmd': u'/bin/oc create -f /tmp/brokerout-5sgEV3 -n default', u'returncode': 1, u'results': {}, u'stderr': u'Error from server (InternalError): error when creating "/tmp/brokerout-5sgEV3": an error on the server ("Error: \'x509: certificate signed by unknown authority (possibly because of \\"crypto/rsa: verification error\\" while trying to verify candidate authority certificate \\"service-catalog-signer\\")\'\\nTrying to reach: \'https://172.30.71.184:443/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers\'") has prevented the request from succeeding (post clusterservicebrokers.servicecatalog.k8s.io)\n', u'stdout': u''}



Version-Release number of selected component (if applicable):
openshift-ansible-playbooks-3.9.33-1.git.56.19ba16e.el7.noarch
ansible-2.4.4.0-1.el7ae.noarch

Comment 6 Fabian von Feilitzsch 2018-08-02 23:53:42 UTC
Did the workaround work?

Comment 9 Jason Montleon 2018-10-17 18:13:29 UTC
Vladislav does the workaround mentioned above get the customer around the issue?

Comment 10 Vladislav Walek 2018-11-16 08:44:44 UTC
Jason, yes it worked for them.
Thx

Comment 11 John Matthews 2018-11-26 17:10:14 UTC
Realigning to Service Catalog so Jay can determine if any work is still required to address this for OCP 4.0.

Jay, this issue looks to be a duplicate of an issue fixed in 3.7.z:
https://bugzilla.redhat.com/show_bug.cgi?id=1523625 

Customers are having success following the workaround identified here:
https://bugzilla.redhat.com/show_bug.cgi?id=1523625#c37

Comment 12 Jay Boyd 2018-11-28 18:30:32 UTC
In 4.0 Service Catalog is installed by a CSV in the Operator Framework.  The SSL Cert is created by the OpenShift Service Serving Cert Signer and OLM registers the API Service.  I believe the CA Bundle will be updated automatically but I'm checking with Evan.

Comment 13 Jay Boyd 2018-12-03 20:13:38 UTC
Evan confirmed with the following:  "The support for apiservices in OLM does not currently use the service serving cert signer, but it does automatically configure and automatically rotate certs for the apiservices. In the future we plan to look at transitioning to service serving cert signer (but if we do that will be transparent)."

Comment 16 Jian Zhang 2019-01-12 04:41:28 UTC
Looks good to me for 4.0, verify it.

The OCP 4.0 version:
mac:jian3 jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-01-12-000105   True        False         10m       Cluster version is 4.0.0-0.nightly-2019-01-12-000105

The OLM version:
mac:jian3 jianzhang$ oc exec olm-operator-6b444f9df7-t7cfs -- olm -version
OLM version: 0.8.0
git commit: 1152f1b

The ServiceCatalog version:
mac:aws-ocp jianzhang$ oc exec controller-manager-64b8dd67d-4wmj9 -- service-catalog --version
v4.0.0-v0.1.38+abebed4-4-dirty;Upstream:v0.1.38

Detailed steps for now:
1, mac:jian3 jianzhang$ oc adm new-project kube-service-catalog
Created project kube-service-catalog

2, $ mac:aws-ocp jianzhang$ cat og-all.yaml 
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: catalog-operators
  namespace: kube-service-catalog
spec:
  selector: {}
3, $ oc create -f og-all.yaml
mac:jian3 jianzhang$ oc project kube-service-catalog
Now using project "kube-service-catalog" on server "https://qe-jian3-api.qe.devcluster.openshift.com:6443".
mac:jian3 jianzhang$ oc get operatorgroup
NAME                AGE
catalog-operators   27s

4, Install the Service Catalog. Or you can do this in Webconsole.
mac:aws-ocp jianzhang$ cat svcat.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: svcat-
  namespace: kube-service-catalog
spec:
  source: rh-operators
  name: svcat
  startingCSV: svcat.v0.1.34
  channel: alpha
mac:aws-ocp jianzhang$ oc create -f svcat.yaml
mac:jian3 jianzhang$ oc get  pods
NAME                                 READY     STATUS    RESTARTS   AGE
apiserver-77576b89cc-5tbkc           2/2       Running   0          31m
controller-manager-64b8dd67d-r2675   1/1       Running   3          31m


5, Check the certs.
mac:jian3 jianzhang$ oc get csv svcat.v0.1.34 -o yaml|grep certs
  certsLastUpdated: 2019-01-12T03:29:30Z
  certsRotateAt: 2021-01-10T03:29:30Z

6, Modify the CA:
mac:jian3 jianzhang$ oc edit apiservice v1beta1.servicecatalog.k8s.io
apiservice.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io edited

7, And, we can see the cert/key are rotated.
mac:jian3 jianzhang$ oc get csv svcat.v0.1.34 -o yaml|grep certs
  certsLastUpdated: 2019-01-12T04:05:43Z
  certsRotateAt: 2021-01-10T04:05:41Z

The volume (apiservice-cert) is remounting.
mac:jian3 jianzhang$ oc get pods
NAME                                 READY     STATUS              RESTARTS   AGE
apiserver-77576b89cc-5tbkc           2/2       Running             0          36m
apiserver-8497d46ff7-pv247           0/2       ContainerCreating   0          3s
controller-manager-64b8dd67d-r2675   0/1       CrashLoopBackOff    4          36m

mac:jian3 jianzhang$ oc get pods
NAME                                 READY     STATUS    RESTARTS   AGE
apiserver-8497d46ff7-pv247           2/2       Running   0          11m
controller-manager-64b8dd67d-r2675   1/1       Running   6          48m


8, Before the CA modification, the olmcahash is:
mac:jian3 jianzhang$ oc get secret v1beta1.servicecatalog.k8s.io-cert -o yaml|grep olmcahash
    olmcahash: ce033ac2c69bb1c5390c578eb5251a813f0af974250a6fd39865190389ffb355

After the CA modification, the olmcahash is:
mac:jian3 jianzhang$ oc get secret v1beta1.servicecatalog.k8s.io-cert -o yaml|grep olmcahash
    olmcahash: 85e072c1ac75e884784435a384e13e023349ae08c4f74c9af61d0edb410cb4c0

9, Install a clusterservicebroker:
1) mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-deploy.yaml
deployment.extensions/ups-broker created
mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-svc.yaml
service/ups-broker created
mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-3.7.yaml
clusterservicebroker.servicecatalog.k8s.io/ups-broker created

2) Check the resource provided by this broker:
mac:jian3 jianzhang$ oc get clusterservicebroker
NAME         URL                                                        STATUS    AGE
ups-broker   http://ups-broker.kube-service-catalog.svc.cluster.local   Ready     27s
mac:jian3 jianzhang$ oc get clusterserviceclass
NAME                                   EXTERNAL-NAME                        BROKER       AGE
4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468   user-provided-service                ups-broker   39s
5f6e6cf6-ffdd-425f-a2c7-3c9258ad2468   user-provided-service-single-plan    ups-broker   39s
8a6229d4-239e-4790-ba1f-8367004d0473   user-provided-service-with-schemas   ups-broker   39s

3) Consume the resources. It works well.
mac:jian3 jianzhang$ oc create -f ../ups-instance.yaml 
serviceinstance.servicecatalog.k8s.io/ups-instance-1 created
mac:jian3 jianzhang$ oc get serviceinstance
NAME             CLASS                                       PLAN      STATUS    AGE
ups-instance-1   ClusterServiceClass/user-provided-service   default   Ready     7s

Comment 20 errata-xmlrpc 2019-06-04 10:40:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758