Description of problem: When using asb operator to install asb, it failed at `Verify service catalog is installed` logs of asb operator; Set broker namespace state=present] ******************\r\n\u001b[1;30mtask path: /opt/ansible/roles/automation-broker/tasks/main.yml:3\u001b[0m\n\u001b[0;36mskipping: [localhost] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\u001b[0m\n\r\nTASK [automation-broker : Verify service catalog is installed] *****************\r\n\u001b[1;30mtask path: /opt/ansible/roles/automation-broker/tasks/main.yml:9\u001b[0m\n2019-01-22 07:35:48,962 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,970 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,979 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,979 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"The conditional check '\\\"servicecatalog.k8s.io\\\" in lookup(\\\"k8s\\\", cluster_info=\\\"api_groups\\\")' failed. The error was: unhandled exception occurred while running the lookup plugin 'k8s'. An Error was a <class 'urllib3.exceptions.MaxRetryError'>, original message: HTTPSConnectionPool(host='172.30.0.1', port=443): Max retries exceeded with url: /version (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))\"}\u001b[0m\n\r \nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m\r\n\n","job":"3870356907665767028","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","error":"exit status 2","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:258"} Version-Release number of selected component (if applicable): OLM version: 0.8.1 git commit: a36ed09 Service-catalog: v4.0.0-v0.1.38+abebed4-4-dirty;Upstream:v0.1.38 Cluster version is 4.0.0-0.alpha-2019-01-22-015156 How reproducible: always Steps to Reproduce: 1. install service-catalog from web-console a. create 'kube-service-catalog' namespace and operator group using files in https://github.com/fusor/catbrokers4/tree/master/files/svcat b. Click "Catalog"-> "Operator Hub" -> "Show community operators"-> "svcat oeprator"->'install' -> target to the "service-catalog" OperatorGroup. 2. install asb using files in https://github.com/fusor/catbrokers4/tree/master/files/asb ├── 00-asb-namespace.yaml ├── 01-asb-catalogsource-configmap.yaml ├── 02-asb-catalogsource.yaml ├── 03-asb-operatorgroup.yaml ├── 04-asb-subscription.yaml ├── 05-asb-cr.yaml └── 06-asb-clusterrolebinding.yaml a. create namespace and operator group b. create cm and catalogsource. c. create subscription and wait for the asb operator ready $ oc get pod -n openshift-ansible-service-broker NAME READY STATUS RESTARTS AGE automation-broker-operator-66b644bfb5-qfjtf 1/1 Running 0 2 d. create asb cr. oc create -f 05-asb-cr.yaml oc create -f 06-asb-clusterrolebinding.yaml Actual results: asb not installed successfully. $ oc get pod -n openshift-ansible-service-broker NAME READY STATUS RESTARTS AGE automation-broker-operator-66b644bfb5-qfjtf 1/1 Running 0 2h $ oc logs -f automation-broker-operator-66b644bfb5-qfjtf {"level":"error","ts":1548142549.0631177,"logger":"logging_event_handler","msg":"","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","gvk":"automationbroker.io/v1alpha1, Kind=AutomationBroker","event_type":"runner_on_failed","job":"3870356907665767028","EventData.Task":"Verify service catalog is installed","EventData.TaskArgs":"msg=Service Catalog must be installed, that=[u'\"servicecatalog.k8s.io\" in lookup(\"k8s\", cluster_info=\"api_groups\")']","EventData.FailedTaskPath":"/opt/ansible/roles/automation-broker/tasks/main.yml:9","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"} {"level":"error","ts":1548142549.225746,"logger":"runner","msg":"\u001b[0;34mansible-playbook 2.7.5\u001b[0m\r\n\u001b[0;34m config file = /etc/ansible/ansible.cfg\u001b[0m\r\n\u001b[0;34m configured module search path = [u'/usr/share/ansible/openshift']\u001b[0m\r\n\u001b[0;34m ansible python module location = /usr/lib/python2.7/site-packages/ansible\u001b[0m\r\n\u001b[0;34m executable location = /usr/bin/ansible-playbook\u001b[0m\r\n\u001b[0;34m python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]\u001b[0m\r\n\u001b[0;34mUsing /etc/ansible/ansible.cfg as config file\u001b[0m\r\n\n\u001b[0;34m/tmp/ansible-operator/runner/automationbroker.io/v1alpha1/AutomationBroker/openshift-ansible-service-broker/ansible-service-broker/inventory/hosts did not meet host_list requirements, check plugin documentation if this is unexpected\u001b[0m\r\n\u001b[0;34m/tmp/ansible-operator/runner/automationbroker.io/v1alpha1/AutomationBroker/openshift-ansible-service-broker/ansible-service-broker/inventory/hosts did not meet script requirements, check plugin documentation if this is unexpected\u001b[0m\r\n\n\u001b[0;34m/tmp/ansible-operator/runner/automationbroker.io/v1alpha1/AutomationBroker/openshift-ansible-service-broker/ansible-service-broker/inventory/hosts did not meet script requirements, check plugin documentation if this is unexpected\u001b[0m\n\r\nPLAYBOOK: deploy.yml ***********************************************************\n\u001b[0;34m1 plays in /opt/ansible/deploy.yml\u001b[0m\n\r\nPLAY [automation-broker-operator] **********************************************\n\u001b[0;34mMETA: ran handlers\u001b[0m\n\r\nTASK [Validation] **************************************************************\r\n\u001b[1;30mtask path: /opt/ansible/deploy.yml:14\u001b[0m\n\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m \"changed\": false, \u001b[0m\r\n\u001b[0;32m \"msg\": \"All assertions passed\"\u001b[0m\r\n\u001b[0;32m}\u001b[0m\ n\r\nTASK [Run automation-broker role] **********************************************\r\n\u001b[1;30mtask path: /opt/ansible/deploy.yml:21\u001b[0m\n\u001b[0;34mstatically imported: /opt/ansible/roles/automation-broker/tasks/build_config.yml\u001b[0m\n\r\nTASK [automation-broker : Set broker namespace state=present] ******************\r\n\u001b[1;30mtask path: /opt/ansible/roles/automation-broker/tasks/main.yml:3\u001b[0m\n\u001b[0;36mskipping: [localhost] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\u001b[0m\n\r\nTASK [automation-broker : Verify service catalog is installed] *****************\r\n\u001b[1;30mtask path: /opt/ansible/roles/automation-broker/tasks/main.yml:9\u001b[0m\n2019-01-22 07:35:48,962 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,970 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,979 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\r\n\n2019-01-22 07:35:48,979 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /version\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"The conditional check '\\\"servicecatalog.k8s.io\\\" in lookup(\\\"k8s\\\", cluster_info=\\\"api_groups\\\")' failed. The error was: An unhandled exception occurred while running the lookup plugin 'k8s'. Error wa s a <class 'urllib3.exceptions.MaxRetryError'>, original message: HTTPSConnectionPool(host='172.30.0.1', port=443): Max retries exceeded with url: /version (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))\"}\u001b[0m\n\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=1 \u001b[0m changed=0 unreachable=0 \u001b[0;31mfailed=1 \u001b[0m\r\n\n","job":"3870356907665767028","name":"ansible-service-broker","namespace":"openshift-ansible-service-broker","error":"exit status 2","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/runner.(*runner).Run.func1\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/pkg/ansible/runner/runner.go:258"} checking service-catalog pod, pods are running. logs of service-catalog api-server pod: $ oc logs -f apiserver-69c7c75b59-s5qg4 -c apiserver -n kube-service-catalog ... I0122 08:22:21.016748 1 wrap.go:42] GET /healthz: (630.04µs) 200 [kube-probe/1.11+ 10.131.0.1:43930] E0122 08:22:22.459006 1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid I0122 08:22:22.459133 1 wrap.go:42] GET /: (207.853µs) 401 [Go-http-client/2.0 10.128.0.1:54860] E0122 08:22:22.463939 1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid I0122 08:22:22.464004 1 wrap.go:42] GET /: (124.01µs) 401 [Go-http-client/2.0 10.128.0.1:54860] E0122 08:22:22.464085 1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid I0122 08:22:22.464175 1 wrap.go:42] GET /: (134.01µs) 401 [Go-http-client/2.0 10.128.0.1:54860] E0122 08:22:22.486403 1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid ... Expected results: install asb successfully. Additional info:
Adding TestBlocker keyword since it is blocking automation broker testing
Install templateservicebroker, also hit this issue. $ oc get templateservicebroker -o yaml -n openshift-template-service-broker apiVersion: v1 items: - apiVersion: osb.openshift.io/v1alpha1 kind: TemplateServiceBroker metadata: creationTimestamp: 2019-01-28T03:02:54Z finalizers: - finalizer.osb.openshift.io generation: 1 name: template-service-broker namespace: openshift-template-service-broker resourceVersion: "38123" selfLink: /apis/osb.openshift.io/v1alpha1/namespaces/openshift-template-service-broker/templateservicebrokers/template-service-broker uid: 34c248fd-22a9-11e9-9f04-060ee74b6582 spec: {} status: conditions: - ansibleResult: changed: 1 completion: 2019-01-28T03:03:02.349635 failures: 1 ok: 4 skipped: 0 lastTransitionTime: 2019-01-28T03:03:02Z message: 'An unhandled exception occurred while running the lookup plugin ''k8s''. Error was a <class ''urllib3.exceptions.MaxRetryError''>, original message: HTTPSConnectionPool(host=''172.30.0.1'', port=443): Max retries exceeded with url: /version (Caused by SSLError(SSLError(1, u''[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)''),))' reason: Failed status: "True" type: Failure - lastTransitionTime: 2019-01-28T03:10:50Z message: Running reconciliation reason: Running status: "False" type: Running kind: List metadata: resourceVersion: "" selfLink: ""
This is a problem with ca.crt that is delivered to the operator. This is being tracked here: https://jira.coreos.com/browse/AUTH-235
In 4.0.0-0.nightly-2019-02-17-024922, I didn't hit this, remove TestBlocker.
When install ASB, I haven't hit it in the recent builds, but TSB still install failed with this error: message: 'An unhandled exception occurred while running the lookup plugin ''template''. Error was a <class ''ansible.errors.AnsibleError''>, original message: An unhandled exception occurred while running the lookup plugin ''k8s''. Error was a <class ''urllib3.exceptions.MaxRetryError''>, original message: HTTPSConnectionPool(host=''172.30.0.1'', port=443): Max retries exceeded with url: /version (Caused by SSLError(SSLError(1, u''[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)''),))'
This is still an auth team bug. Please re-assign as discussed.
As discussed, resign to AUTH, related jira task : https://jira.coreos.com/browse/AUTH-235
TSB install still hit this issue, move to ASSIGNED.
Which version of the installer are you using? It's likely you're missing the patch.
(In reply to Erica von Buelow from comment #13) > Which version of the installer are you using? It's likely you're missing the > patch. With my last env, the installer version is : v4.0.0-0.174.0.0-dirty, OCP Version is: 4.0.0-0.nightly-2019-02-17-024922
With installer: v4.0.0-0.177.0.1-dirty Cluster version is 4.0.0-0.nightly-2019-02-20-194410 this issue is fixed. Thanks
*** Bug 1670282 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days