Please supply me with the following information, I am having a hard time determining the environment you are testing under. 1) OpenShift version used for testing? 2) Broker version used for testing? (it should be present in the broker log file)
Using 3.9 I was able to recreate the original error. 2019-02-20T16:39:50.88Z] [DEBUG] - Dao::BatchGetRaw [2019-02-20T16:39:50.881Z] [DEBUG] - Successfully loaded [ 0 ] objects from etcd dir [ /spec ] [2019-02-20T16:39:51.028Z] [WARNING] - registry: 0x1584a70 was unable to complete bootstrap - Get https://registry.access.stage.redhat.com/v1/search?q="*-apb": x509: certificate signed by unknown authority [2019-02-20T16:39:51.028Z] [ERROR] - Failed to bootstrap on startup! [2019-02-20T16:39:51.028Z] [ERROR] - all registries failed on bootstrap time="2019-02-20T16:39:51Z" level=error msg="unable to retrieve image names for registry rh - Get https://registry.access.stage.redhat.com/v1/search?q=\"*-apb\": x509: certificate signed by unknown authority" I will investigate.
We are back porting the ability to skip TLS verification which is how we deal with internal registries like this without having the certificate. It is a configuration item.
Jesus, Is it ready for now? If yes, could you tell me how to config it? Thanks!
Patch posted to 3.9 branch: https://github.com/openshift/ansible-service-broker/pull/1194 We backported the SkipVerifyTLS option from 3.11 back to 3.9. It will be ready in a couple of days. Once it gets reviewed, merged. We can rebuild the packages. Then rebuild the images for QE.
Jesus, do we need to doc this changes (backported the SkipVerifyTLS option from 3.11 back to 3.9) in related release doc?
Probably not. I just removed it. Makes things simpler too.
Could please take high priority to fix this? in v3.9, only stage registry can be used as downstream registry. This blocks errata testing. Thanks.
I've retested and it works for me using `http://registry.access.stage.redhat.com` and `skip_verify_tls: true`. Broker version 1.1.20 being used. ============================================================ == Starting Ansible Service Broker... == ============================================================ [2019-07-02T19:31:16.971Z] [INFO] - Ansible Service Broker Version: 1.1.20 [2019-07-02T19:31:16.971Z] [NOTICE] - Initializing clients... [2019-07-02T19:31:16.972Z] [DEBUG] - Trying to connect to etcd [2019-07-02T19:31:16.972Z] [INFO] - == ETCD CX == [2019-07-02T19:31:16.972Z] [INFO] - EtcdHost: asb-etcd.ansible-service-broker.svc [2019-07-02T19:31:16.972Z] [INFO] - EtcdPort: 2379 [2019-07-02T19:31:16.972Z] [INFO] - Endpoints: [https://asb-etcd.ansible-service-broker.svc:2379] [2019-07-02T19:31:16.991Z] [INFO] - Etcd Version [Server: 3.3.8, Cluster: 3.3.0] [2019-07-02T19:31:16.996Z] [DEBUG] - Connecting to Cluster [2019-07-02T19:31:17.006Z] [INFO] - OpenShift version: v3.9.0+71543b2-33 [2019-07-02T19:31:17.009Z] [DEBUG] - plugin for the network - [2019-07-02T19:31:17.009Z] [NOTICE] - unable to retrieve the network plugin, defaulting to not joining networks - clusternetworks.network.openshift.io "default" not found [2019-07-02T19:31:17.009Z] [INFO] - Kubernetes version: v1.9.1+a0ce1bc657 [2019-07-02T19:31:17.009Z] [DEBUG] - Connecting Dao [2019-07-02T19:31:17.009Z] [DEBUG] - Connecting Registry [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get user from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get pass from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get org from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get images from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get namespaces from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get fail_on_error from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get black_list from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get auth_type from config [2019-07-02T19:31:17.01Z] [DEBUG] - Unable to get auth_name from config [2019-07-02T19:31:17.01Z] [INFO] - == REGISTRY CX == [2019-07-02T19:31:17.01Z] [INFO] - Name: rh [2019-07-02T19:31:17.01Z] [INFO] - Type: rhcc [2019-07-02T19:31:17.01Z] [INFO] - Url: http://registry.access.stage.redhat.com [2019-07-02T19:31:17.01Z] [WARNING] - skipping verification of registry TLS certificate per adapter configuration [2019-07-02T19:31:17.01Z] [DEBUG] - Creating filter for registry: rh [2019-07-02T19:31:17.01Z] [DEBUG] - whitelist: [.*-apb$] [2019-07-02T19:31:17.01Z] [DEBUG] - blacklist: [] [2019-07-02T19:31:17.01Z] [DEBUG] - Initializing WorkEngine The broker sees the following APBs in the list [2019-07-02T19:31:17.931Z] [DEBUG] - Filter applied against registry: rh [2019-07-02T19:31:17.931Z] [DEBUG] - APBs passing white/blacklist filter: [2019-07-02T19:31:17.932Z] [DEBUG] - -> rhv4-tech-preview/ovirt-flexvolume-driver-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> mobile-1-tech-preview/mobile-app-metrics-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> openshift3/mariadb-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> openshift4/mariadb-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> mobile-1-tech-preview/mobile-sync-app-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> mobile-1-tech-preview/mobile-unifiedpush-server-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> openshift4/mediawiki-apb [2019-07-02T19:31:17.932Z] [DEBUG] - -> mobile-1-tech-preview/mobile-developer-console-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> mobile-1-tech-preview/mobile-identity-management-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift4/postgres-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> codeready-workspaces-beta/server-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift4/mysql-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> rhpam-7/rhpam-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift3/automation-broker-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift3/mediawiki-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> cnv-tech-preview/kubevirt-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> cnv-tech-preview/import-vm-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift3/mysql-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift3/postgresql-apb [2019-07-02T19:31:17.933Z] [DEBUG] - -> openshift4/postgresql-apb Only a few are available, manyof the image urls return 404 Not Found. But that is a function of how the registry is filled not a broker issue. I have attached my broker-config and my broker log.
Created attachment 1586856 [details] broker config Here is my broker configmap you can see the registry entry defined within. registry: - type: "rhcc" name: "rh" url: "http://registry.access.stage.redhat.com" org: tag: "v3.9" skip_verify_tls: true white_list: - ".*-apb$"
Created attachment 1586857 [details] my broker logs This log shows the broker comes up and connects to the stage registry with the skip_verify_tls set to true.
@Jesue, your openshift version is v3.9.0+71543b2-33, it is too old. my cluster is: OpenShift version: v3.9.85 I try this again, the error is different again. My registry: registry: - type: rhcc name: rh url: https://registry.access.stage.redhat.com org: tag: v3.9 white_list: [.*-apb$] skip_verify_tls: true auth_type: "" auth_name: "" The asb log: [2019-07-03T03:34:02.658Z] [ERROR] - Image 'registry.access.stage.redhat.com/openshift4/mysql-apb:v3.9' may not exist in registry. [2019-07-03T03:34:02.658Z] [ERROR] - {"errors":[{"code":"UNSUPPORTED","message":"This repo requires terms acceptance and is only available on registry.redhat.io"}]} [2019-07-03T03:34:02.823Z] [ERROR] - Image 'registry.access.stage.redhat.com/openshift4/mediawiki-apb:v3.9' may not exist in registry. It should check image in .../openshift3/... not /openshift4/... and I config `org` to `openshift3` in registry: - type: rhcc name: rh url: https://registry.access.stage.redhat.com org: openshift3 tag: v3.9 white_list: [.*-apb$] skip_verify_tls: true auth_type: "" auth_name: "" but the org parameter not work for rhcc registry, still got the following error: [2019-07-03T03:52:32.813Z] [ERROR] - Image 'registry.access.stage.redhat.com/openshift4/mariadb-apb:v3.9' may not exist in registry.
Created attachment 1586912 [details] asb-log in v3.9.85
Add whitelist to registry config, it succeeds to use stage registry. registry: - type: rhcc name: rh url: https://registry.access.stage.redhat.com org: tag: v3.9.85 white_list: - openshift3/mediawiki-apb - openshift3/postgresql-apb - openshift3/mariadb-apb - openshift3/mysql-apb skip_verify_tls: true auth_type: "" auth_name: "" asb log: oc logs -f asb-1-pnzr8 | grep 'into Spec' [2019-07-03T06:06:57.207Z] [DEBUG] - Successfully converted Image registry.access.stage.redhat.com/openshift3/mariadb-apb:v3.9.85 into Spec [2019-07-03T06:06:57.332Z] [DEBUG] - Successfully converted Image registry.access.stage.redhat.com/openshift3/mysql-apb:v3.9.85 into Spec [2019-07-03T06:06:57.454Z] [DEBUG] - Successfully converted Image registry.access.stage.redhat.com/openshift3/mediawiki-apb:v3.9.85 into Spec [2019-07-03T06:06:57.578Z] [DEBUG] - Successfully converted Image registry.access.stage.redhat.com/openshift3/postgresql-apb:v3.9.85 into Spec
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1757