Bug 1603343 - "Error: 'x509: certificate signed by unknown authority" at "TASK [ansible_service_broker : Create the Broker resource in the catalog]"
Summary: "Error: 'x509: certificate signed by unknown authority" at "TASK [ansible_ser...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Catalog
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Dan Geoffroy
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-19 14:29 UTC by François Cami
Modified: 2019-06-04 10:40 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2019-06-04 10:40:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:40:28 UTC

Description François Cami 2018-07-19 14:29:11 UTC
Description of problem:

     Play:     Upgrade Service Catalog
     Task:     Create the Broker resource in the catalog
     Message:  {u'cmd': u'/bin/oc create -f /tmp/brokerout-5sgEV3 -n default', u'returncode': 1, u'results': {}, u'stderr': u'Error from server (InternalError): error when creating "/tmp/brokerout-5sgEV3": an error on the server ("Error: \'x509: certificate signed by unknown authority (possibly because of \\"crypto/rsa: verification error\\" while trying to verify candidate authority certificate \\"service-catalog-signer\\")\'\\nTrying to reach: \'https://172.30.71.184:443/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers\'") has prevented the request from succeeding (post clusterservicebrokers.servicecatalog.k8s.io)\n', u'stdout': u''}



Version-Release number of selected component (if applicable):
openshift-ansible-playbooks-3.9.33-1.git.56.19ba16e.el7.noarch
ansible-2.4.4.0-1.el7ae.noarch

Comment 6 Fabian von Feilitzsch 2018-08-02 23:53:42 UTC
Did the workaround work?

Comment 9 Jason Montleon 2018-10-17 18:13:29 UTC
Vladislav does the workaround mentioned above get the customer around the issue?

Comment 10 Vladislav Walek 2018-11-16 08:44:44 UTC
Jason, yes it worked for them.
Thx

Comment 11 John Matthews 2018-11-26 17:10:14 UTC
Realigning to Service Catalog so Jay can determine if any work is still required to address this for OCP 4.0.

Jay, this issue looks to be a duplicate of an issue fixed in 3.7.z:
https://bugzilla.redhat.com/show_bug.cgi?id=1523625 

Customers are having success following the workaround identified here:
https://bugzilla.redhat.com/show_bug.cgi?id=1523625#c37

Comment 12 Jay Boyd 2018-11-28 18:30:32 UTC
In 4.0 Service Catalog is installed by a CSV in the Operator Framework.  The SSL Cert is created by the OpenShift Service Serving Cert Signer and OLM registers the API Service.  I believe the CA Bundle will be updated automatically but I'm checking with Evan.

Comment 13 Jay Boyd 2018-12-03 20:13:38 UTC
Evan confirmed with the following:  "The support for apiservices in OLM does not currently use the service serving cert signer, but it does automatically configure and automatically rotate certs for the apiservices. In the future we plan to look at transitioning to service serving cert signer (but if we do that will be transparent)."

Comment 16 Jian Zhang 2019-01-12 04:41:28 UTC
Looks good to me for 4.0, verify it.

The OCP 4.0 version:
mac:jian3 jianzhang$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-01-12-000105   True        False         10m       Cluster version is 4.0.0-0.nightly-2019-01-12-000105

The OLM version:
mac:jian3 jianzhang$ oc exec olm-operator-6b444f9df7-t7cfs -- olm -version
OLM version: 0.8.0
git commit: 1152f1b

The ServiceCatalog version:
mac:aws-ocp jianzhang$ oc exec controller-manager-64b8dd67d-4wmj9 -- service-catalog --version
v4.0.0-v0.1.38+abebed4-4-dirty;Upstream:v0.1.38

Detailed steps for now:
1, mac:jian3 jianzhang$ oc adm new-project kube-service-catalog
Created project kube-service-catalog

2, $ mac:aws-ocp jianzhang$ cat og-all.yaml 
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: catalog-operators
  namespace: kube-service-catalog
spec:
  selector: {}
3, $ oc create -f og-all.yaml
mac:jian3 jianzhang$ oc project kube-service-catalog
Now using project "kube-service-catalog" on server "https://qe-jian3-api.qe.devcluster.openshift.com:6443".
mac:jian3 jianzhang$ oc get operatorgroup
NAME                AGE
catalog-operators   27s

4, Install the Service Catalog. Or you can do this in Webconsole.
mac:aws-ocp jianzhang$ cat svcat.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  generateName: svcat-
  namespace: kube-service-catalog
spec:
  source: rh-operators
  name: svcat
  startingCSV: svcat.v0.1.34
  channel: alpha
mac:aws-ocp jianzhang$ oc create -f svcat.yaml
mac:jian3 jianzhang$ oc get  pods
NAME                                 READY     STATUS    RESTARTS   AGE
apiserver-77576b89cc-5tbkc           2/2       Running   0          31m
controller-manager-64b8dd67d-r2675   1/1       Running   3          31m


5, Check the certs.
mac:jian3 jianzhang$ oc get csv svcat.v0.1.34 -o yaml|grep certs
  certsLastUpdated: 2019-01-12T03:29:30Z
  certsRotateAt: 2021-01-10T03:29:30Z

6, Modify the CA:
mac:jian3 jianzhang$ oc edit apiservice v1beta1.servicecatalog.k8s.io
apiservice.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io edited

7, And, we can see the cert/key are rotated.
mac:jian3 jianzhang$ oc get csv svcat.v0.1.34 -o yaml|grep certs
  certsLastUpdated: 2019-01-12T04:05:43Z
  certsRotateAt: 2021-01-10T04:05:41Z

The volume (apiservice-cert) is remounting.
mac:jian3 jianzhang$ oc get pods
NAME                                 READY     STATUS              RESTARTS   AGE
apiserver-77576b89cc-5tbkc           2/2       Running             0          36m
apiserver-8497d46ff7-pv247           0/2       ContainerCreating   0          3s
controller-manager-64b8dd67d-r2675   0/1       CrashLoopBackOff    4          36m

mac:jian3 jianzhang$ oc get pods
NAME                                 READY     STATUS    RESTARTS   AGE
apiserver-8497d46ff7-pv247           2/2       Running   0          11m
controller-manager-64b8dd67d-r2675   1/1       Running   6          48m


8, Before the CA modification, the olmcahash is:
mac:jian3 jianzhang$ oc get secret v1beta1.servicecatalog.k8s.io-cert -o yaml|grep olmcahash
    olmcahash: ce033ac2c69bb1c5390c578eb5251a813f0af974250a6fd39865190389ffb355

After the CA modification, the olmcahash is:
mac:jian3 jianzhang$ oc get secret v1beta1.servicecatalog.k8s.io-cert -o yaml|grep olmcahash
    olmcahash: 85e072c1ac75e884784435a384e13e023349ae08c4f74c9af61d0edb410cb4c0

9, Install a clusterservicebroker:
1) mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-deploy.yaml
deployment.extensions/ups-broker created
mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-svc.yaml
service/ups-broker created
mac:jian3 jianzhang$ oc create -f https://raw.githubusercontent.com/jianzhangbjz/v3-testfiles/master/svc-catalog/ups-broker-3.7.yaml
clusterservicebroker.servicecatalog.k8s.io/ups-broker created

2) Check the resource provided by this broker:
mac:jian3 jianzhang$ oc get clusterservicebroker
NAME         URL                                                        STATUS    AGE
ups-broker   http://ups-broker.kube-service-catalog.svc.cluster.local   Ready     27s
mac:jian3 jianzhang$ oc get clusterserviceclass
NAME                                   EXTERNAL-NAME                        BROKER       AGE
4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468   user-provided-service                ups-broker   39s
5f6e6cf6-ffdd-425f-a2c7-3c9258ad2468   user-provided-service-single-plan    ups-broker   39s
8a6229d4-239e-4790-ba1f-8367004d0473   user-provided-service-with-schemas   ups-broker   39s

3) Consume the resources. It works well.
mac:jian3 jianzhang$ oc create -f ../ups-instance.yaml 
serviceinstance.servicecatalog.k8s.io/ups-instance-1 created
mac:jian3 jianzhang$ oc get serviceinstance
NAME             CLASS                                       PLAN      STATUS    AGE
ups-instance-1   ClusterServiceClass/user-provided-service   default   Ready     7s

Comment 20 errata-xmlrpc 2019-06-04 10:40:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.