Description of problem: After complete install by openshift-ansilbe with service-catalog and asb. Then get serviceclass is empty. When remove/add apb image on registry, asb can't auto update apb spec info. Now we must invoke bootstrap api by manual, So when create broker resource, we can't get latest services. Version-Release number of selected component (if applicable): 0.9.4 How reproducible: Always Steps to Reproduce: 1. Start asb with configure dockerhub 2. Get catalog # curl http://172.30.55.244:1338/v2/catalog { "services": [] } 3. Push some images to registry org then wait long time get catalog again # curl http://172.30.55.244:1338/v2/catalog { "services": [] } Actual results: Expected results: After image update on registry, asb should auto update spec. Additional info: 4. Manual invoke bootstrap and get catalog # curl -H 'X-Broker-API-Version: 2.9' -X POST -v http://172.30.55.244:1338/v2/bootstrap * About to connect() to 172.30.55.244 port 1338 (#0) * Trying 172.30.55.244... * Connected to 172.30.55.244 (172.30.55.244) port 1338 (#0) > POST /v2/bootstrap HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 172.30.55.244:1338 > Accept: */* > X-Broker-API-Version: 2.9 > < HTTP/1.1 200 OK < Content-Type: application/json < Date: Thu, 06 Jul 2017 07:14:00 GMT < Content-Length: 42 < { "spec_count": 1, "image_count": 1 } * Connection #0 to host 172.30.55.244 left intact # curl http://172.30.55.244:1338/v2/catalog { "services": [ { "name": "mediawiki123-apb", "id": "55c53a5d-65a6-4c27-88fc-e027410b1337", "description": "Mediawiki123 apb implementation", "bindable": false, "metadata": { "console.openshift.io/iconClass": "icon-mediawiki", "displayName": "Mediawiki (APB)", "documentationUrl": "https://www.mediawiki.org/wiki/Documentation", "longDescription": "An apb that deploys Mediawiki 1.23", "parameters": [ { "mediawiki_db_schema": { "title": "Mediawiki DB Schema", "type": "string", "default": "mediawiki" } }, { "mediawiki_site_name": { "title": "Mediawiki Site Name", "type": "string", "default": "MediaWiki" } }, { "mediawiki_site_lang": { "title": "Mediawiki Site Language", "type": "string", "default": "en" } }, { "mediawiki_admin_user": { "title": "Mediawiki Admin User", "type": "string", "default": "admin" } }, { "mediawiki_admin_pass": { "title": "Mediawiki Admin User Password", "type": "string" } } ] }, "plans": [ { "id": "4c10ff42-be89-420a-9bab-27a9bef9aed8", "name": "default", "description": "Default plan", "free": true, "schemas": { "service_instance": { "create": { "parameters": { "$schema": "http://json-schema.org/draft-04/schema", "additionalProperties": false, "properties": { "mediawiki_admin_pass": { "title": "Mediawiki Admin User Password", "type": "string" }, "mediawiki_admin_user": { "default": "admin", "title": "Mediawiki Admin User", "type": "string" }, "mediawiki_db_schema": { "default": "mediawiki", "title": "Mediawiki DB Schema", "type": "string" }, "mediawiki_site_lang": { "default": "en", "title": "Mediawiki Site Language", "type": "string" }, "mediawiki_site_name": { "default": "MediaWiki", "title": "Mediawiki Site Name", "type": "string" } }, "required": [ "mediawiki_db_schema", "mediawiki_site_name", "mediawiki_site_lang", "mediawiki_admin_user", "mediawiki_admin_pass" ], "type": "object" } }, "update": {} }, "service_binding": { "create": { "parameters": { "$schema": "http://json-schema.org/draft-04/schema", "additionalProperties": false, "properties": { "mediawiki_admin_pass": { "title": "Mediawiki Admin User Password", "type": "string" }, "mediawiki_admin_user": { "default": "admin", "title": "Mediawiki Admin User", "type": "string" }, "mediawiki_db_schema": { "default": "mediawiki", "title": "Mediawiki DB Schema", "type": "string" }, "mediawiki_site_lang": { "default": "en", "title": "Mediawiki Site Language", "type": "string" }, "mediawiki_site_name": { "default": "MediaWiki", "title": "Mediawiki Site Name", "type": "string" } }, "type": "object" } } } } } ] } ] }
Aligning to 3.7.0 Note for 3.6.0 we only intended to support 2 APBs from registry.access.redhat.com, so I don't consider this applicable for 3.6.0, but it's a good feature for us to implement in next release.
Now after enable service-catalog, asb. There will be create a broker named 'ansible-service-broker'. But when get serviceclass there is nothing. At least we need trigger bootstrap in openshift-ansible script. [root@ip-172-18-6-17 ~]# oc get broker NAME KIND ansible-service-broker Broker.v1alpha1.servicecatalog.k8s.io [root@ip-172-18-6-17 ~]# oc get serviceclass No resources found. Let custome manually trigger bootstrap after install is not good user experience. How do you think?
DeShuai, Agreed we need a mechanism so user does not need to trigger the bootstrap manually. There was some confusion interpreting this BZ, we thought the BZ was asking for Broker to watch the registry and detect new APBs as they are published, that is desired but outside of 3.6 scope. Agree for 3.6 the Broker needs to read the registry initially and see the APBs which are published. I'll move this back to 3.6.0 and we will address shortly.
https://github.com/openshift/ansible-service-broker/pull/265
I get below error: [2017-07-19T09:35:58.072Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: [2017-07-19T09:35:58.072Z] [WARNING] invalid character '<' looking for beginning of value [root@host-8-175-72 ~]# oc get configmap broker-config -o yaml apiVersion: v1 data: broker-config: | registry: name: "rhcc" url: "xxxxxx" -> use our stage registry user: "" pass: "" org: "openshift3" dao: etcd_host: etcd etcd_port: 2379 log: logfile: /var/log/ansible-service-broker/asb.log stdout: true level: "info" color: true openshift: {} broker: dev_broker: false launch_apb_on_bind: false bootstrap_on_startup: true recovery: true output_request: false kind: ConfigMap metadata: creationTimestamp: 2017-07-19T06:29:06Z labels: app: ansible-service-broker name: broker-config namespace: openshift-ansible-service-broker resourceVersion: "13906" selfLink: /api/v1/namespaces/openshift-ansible-service-broker/configmaps/broker-config uid: 90f52d80-6c4b-11e7-a0c3-fa163ec489fc [root@host-8-175-72 ~]# oc get po NAME READY STATUS RESTARTS AGE asb-1222154024-w8p08 1/1 Running 0 1m etcd-3119139199-jlhlc 1/1 Running 2 6m [root@host-8-175-72 ~]# oc logs asb-1222154024-w8p08 Using config file mounted to /etc/ansible-service-broker/config.yaml ============================================================ == Starting Ansible Service Broker... == ============================================================ [2017-07-19T09:35:54.807Z] [INFO] == ETCD CX == [2017-07-19T09:35:54.807Z] [INFO] EtcdHost: etcd [2017-07-19T09:35:54.807Z] [INFO] EtcdPort: 2379 [2017-07-19T09:35:54.807Z] [INFO] Endpoints: [http://etcd:2379] [2017-07-19T09:35:54.822Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0] [2017-07-19T09:35:54.823Z] [INFO] Cluster Client Created [2017-07-19T09:35:54.869Z] [INFO] Kubernetes version: v1.6.1+5115d708d7 [2017-07-19T09:35:54.869Z] [INFO] == REGISTRY CX == [2017-07-19T09:35:54.869Z] [INFO] Name: rhcc [2017-07-19T09:35:54.869Z] [INFO] Url: registry.access.stage.redhat.com [2017-07-19T09:35:54.876Z] [INFO] Listening for provision messages [2017-07-19T09:35:55.349Z] [INFO] Initiating Recovery Process [2017-07-19T09:35:55.352Z] [INFO] No jobs to recover [2017-07-19T09:35:55.352Z] [NOTICE] [2017-07-19T09:35:55.352Z] [INFO] Broker configured to bootstrap on startup [2017-07-19T09:35:55.352Z] [INFO] Attempting bootstrap... [2017-07-19T09:35:55.352Z] [INFO] AnsibleBroker::Bootstrap [2017-07-19T09:35:58.072Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: [2017-07-19T09:35:58.072Z] [WARNING] invalid character '<' looking for beginning of value [2017-07-19T09:35:58.072Z] [NOTICE] Broker successfully bootstrapped on startup [2017-07-19T09:35:58.072Z] [NOTICE] Ansible Service Broker Started [2017-07-19T09:35:58.072Z] [NOTICE] Listening on http://0.0.0.0:1338 [2017-07-19T09:36:11.023Z] [INFO] AnsibleBroker::Catalog 10.129.0.1 - - [19/Jul/2017:09:36:11 +0000] "GET /v2/catalog HTTP/1.1" 200 21 [2017-07-19T09:36:25.811Z] [INFO] AnsibleBroker::Catalog 10.129.0.1 - - [19/Jul/2017:09:36:25 +0000] "GET /v2/catalog HTTP/1.1" 200 21 [2017-07-19T09:36:49.291Z] [INFO] AnsibleBroker::Catalog 10.129.0.1 - - [19/Jul/2017:09:36:49 +0000] "GET /v2/catalog HTTP/1.1" 200 21
Checked again and found the issue can not be reproduced now, and bootstrap_on_startup work well. # oc logs -f asb-1222154024-phdgn Using config file mounted to /etc/ansible-service-broker/config.yaml ============================================================ == Starting Ansible Service Broker... == ============================================================ [2017-07-20T06:04:07.195Z] [INFO] == ETCD CX == [2017-07-20T06:04:07.195Z] [INFO] EtcdHost: etcd [2017-07-20T06:04:07.195Z] [INFO] EtcdPort: 2379 [2017-07-20T06:04:07.195Z] [INFO] Endpoints: [http://etcd:2379] [2017-07-20T06:04:07.216Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0] [2017-07-20T06:04:07.217Z] [INFO] Cluster Client Created [2017-07-20T06:04:07.244Z] [INFO] Kubernetes version: v1.6.1+5115d708d7 [2017-07-20T06:04:07.244Z] [INFO] == REGISTRY CX == [2017-07-20T06:04:07.244Z] [INFO] Name: rhcc [2017-07-20T06:04:07.244Z] [INFO] Url: registry.access.stage.redhat.com [2017-07-20T06:04:07.245Z] [INFO] Listening for provision messages [2017-07-20T06:04:07.732Z] [INFO] Initiating Recovery Process [2017-07-20T06:04:07.734Z] [INFO] Recovery complete [2017-07-20T06:04:07.734Z] [NOTICE] recover called [2017-07-20T06:04:07.734Z] [INFO] Broker configured to bootstrap on startup [2017-07-20T06:04:07.734Z] [INFO] Attempting bootstrap... [2017-07-20T06:04:07.734Z] [INFO] AnsibleBroker::Bootstrap [2017-07-20T06:04:12.125Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping. [2017-07-20T06:04:12.462Z] [NOTICE] Broker successfully bootstrapped on startup [2017-07-20T06:04:12.462Z] [NOTICE] Ansible Service Broker Started [2017-07-20T06:04:12.462Z] [NOTICE] Listening on http://0.0.0.0:1338 [2017-07-20T06:04:24.468Z] [INFO] AnsibleBroker::Catalog # apb list --broker http://asb-1338-openshift-ansible-service-broker.0719-92s.qe.rhcloud.com ID NAME DESCRIPTION e1bcd4a8-ce02-4585-9f4c-18abd56d76f2 postgresql-apb SCL PostgreSQL apb implementation 4fbcc051-03af-40c7-86e2-dc12977e6b5d mediawiki-apb Mediawiki123 apb implementation
QE Please re-test, let us know if you require anything further to close this issue out. The issue seen in comment #5 we think is due to an error from stage registry when it returned html data instead of json content. Perhaps it's related to configuring with https vs http for the stage repo.
The issue is related to bad response from stage registry. Configured with http for stage ansible_service_broker_registry_url: "http://registry.access.stage.redhat.com" Saw error: [2017-07-24T19:08:56.241Z] [INFO] Broker configured to bootstrap on startupESC[0m [2017-07-24T19:08:56.241Z] [INFO] Attempting bootstrap...ESC[0m [2017-07-24T19:08:56.241Z] [INFO] AnsibleBroker::BootstrapESC[0m ESC[36m[2017-07-24T19:08:56.241Z] [DEBUG] Dao::BatchGetRawESC[0m ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] RHCCRegistry::LoadSpecsESC[0m ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] RHCCRegistry::LoadImagesESC[0m ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] Using http://registry.access.stage.redhat.com to source APB images using query:"*-apb"ESC[0m ESC[36m[2017-07-24T19:08:56.669Z] [DEBUG] Got Image Response from RHCCESC[0m ESC[33m[2017-07-24T19:08:56.669Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: ESC[0m ESC[33m[2017-07-24T19:08:56.669Z] [WARNING] invalid character '<' looking for beginning of valueESC[0m ESC[32m[2017-07-24T19:08:56.669Z] [NOTICE] Broker successfully bootstrapped on startupESC[0m ESC[32m[2017-07-24T19:08:56.669Z] [NOTICE] Ansible Service Broker StartedESC[0m ~30 minutes later manually triggered bootstrap and no issues. I did not make any other changes. # curl -X POST http://asb-1338-openshift-ansible-service-broker.apps.192.168.120.4.nip.io/v2/bootstrap { "spec_count": 2, "image_count": 3 } [2017-07-24T19:39:04.239Z] [INFO] AnsibleBroker::Bootstrap [2017-07-24T19:39:04.239Z] [DEBUG] Dao::BatchGetRaw [2017-07-24T19:39:04.24Z] [DEBUG] RHCCRegistry::LoadSpecs [2017-07-24T19:39:04.24Z] [DEBUG] RHCCRegistry::LoadImages [2017-07-24T19:39:04.24Z] [DEBUG] Using http://registry.access.stage.redhat.com to source APB images using query:"*-apb" [2017-07-24T19:39:06.613Z] [DEBUG] Got Image Response from RHCC [2017-07-24T19:39:06.613Z] [DEBUG] Properly unmarshalled image response [2017-07-24T19:39:06.614Z] [DEBUG] Found 3 images in RHCC [2017-07-24T19:39:06.614Z] [DEBUG] RHCCRegistry::imageToSpec [2017-07-24T19:39:06.953Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping. [2017-07-24T19:39:06.953Z] [DEBUG] RHCCRegistry::imageToSpec [2017-07-24T19:39:07.149Z] [DEBUG] Successfully converted RHCC Image postgresql-apb into Spec [2017-07-24T19:39:07.149Z] [DEBUG] RHCCRegistry::imageToSpec [2017-07-24T19:39:07.342Z] [DEBUG] Successfully converted RHCC Image mediawiki-apb into Spec 10.128.0.1 - - [24/Jul/2017:19:39:04 +0000] "POST /v2/bootstrap HTTP/1.1" 200 42
Our understanding of this issue is that sometimes the stage registry will return a bad response. When this happens, the Broker logs a warning that auto bootstrap fails and broker continues. User is left with a service catalog lacking APBs. The cause is not obvious, User needs to look at ASB logs and search for WARNING entries to see that auto bootstrap failed. Next they need to trigger a manual bootstrap and wait for Service Catalog relist the Broker, then APBs are available in WebUI. To improve this experience we will make a change for 3.6.0 that if auto bootstrap fails the broker will fail loudly and stop. Kube will retry the pod, eventually if the response from the registry is good the bootstrap will succeed.
Made the broker fail on error. https://github.com/openshift/ansible-service-broker/pull/300
Checked with ansible-service-broker 0.9.11, and found the bootstrap_on_startup work well, and the pod will stop and kube will retry if bootstrap failed. so verify this issue.