Bug 1468173

Summary: asb need auto bootstrap apb image spec from container catalog
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: Service BrokerAssignee: Fabian von Feilitzsch <fabian>
Status: CLOSED CURRENTRELEASE QA Contact: weiwei jiang <wjiang>
Severity: medium Docs Contact:
Priority: high    
Version: 3.6.0CC: aos-bugs, dma, ernelson, jmatthew, wjiang
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-10 20:54:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description DeShuai Ma 2017-07-06 07:34:08 UTC 
Description of problem:
After complete install by openshift-ansilbe with service-catalog and asb. Then get serviceclass is empty.

When remove/add apb image on registry, asb can't auto update apb spec info. Now we must invoke bootstrap api by manual, So when create broker resource, we can't get latest services.

Version-Release number of selected component (if applicable):
0.9.4

How reproducible:
Always

Steps to Reproduce:
1. Start asb with configure dockerhub
2. Get catalog
# curl http://172.30.55.244:1338/v2/catalog
{
  "services": []
}

3. Push some images to registry org then wait long time get catalog again
# curl http://172.30.55.244:1338/v2/catalog
{
  "services": []
}

Actual results:


Expected results:
After image update on registry, asb should auto update spec.


Additional info:
4. Manual invoke bootstrap and get catalog
# curl -H 'X-Broker-API-Version: 2.9' -X POST -v http://172.30.55.244:1338/v2/bootstrap
* About to connect() to 172.30.55.244 port 1338 (#0)
*   Trying 172.30.55.244...
* Connected to 172.30.55.244 (172.30.55.244) port 1338 (#0)
> POST /v2/bootstrap HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 172.30.55.244:1338
> Accept: */*
> X-Broker-API-Version: 2.9
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Thu, 06 Jul 2017 07:14:00 GMT
< Content-Length: 42
< 
{
  "spec_count": 1,
  "image_count": 1
}
* Connection #0 to host 172.30.55.244 left intact
# curl http://172.30.55.244:1338/v2/catalog
{
  "services": [
    {
      "name": "mediawiki123-apb",
      "id": "55c53a5d-65a6-4c27-88fc-e027410b1337",
      "description": "Mediawiki123 apb implementation",
      "bindable": false,
      "metadata": {
        "console.openshift.io/iconClass": "icon-mediawiki",
        "displayName": "Mediawiki (APB)",
        "documentationUrl": "https://www.mediawiki.org/wiki/Documentation",
        "longDescription": "An apb that deploys Mediawiki 1.23",
        "parameters": [
          {
            "mediawiki_db_schema": {
              "title": "Mediawiki DB Schema",
              "type": "string",
              "default": "mediawiki"
            }
          },
          {
            "mediawiki_site_name": {
              "title": "Mediawiki Site Name",
              "type": "string",
              "default": "MediaWiki"
            }
          },
          {
            "mediawiki_site_lang": {
              "title": "Mediawiki Site Language",
              "type": "string",
              "default": "en"
            }
          },
          {
            "mediawiki_admin_user": {
              "title": "Mediawiki Admin User",
              "type": "string",
              "default": "admin"
            }
          },
          {
            "mediawiki_admin_pass": {
              "title": "Mediawiki Admin User Password",
              "type": "string"
            }
          }
        ]
      },
      "plans": [
        {
          "id": "4c10ff42-be89-420a-9bab-27a9bef9aed8",
          "name": "default",
          "description": "Default plan",
          "free": true,
          "schemas": {
            "service_instance": {
              "create": {
                "parameters": {
                  "$schema": "http://json-schema.org/draft-04/schema",
                  "additionalProperties": false,
                  "properties": {
                    "mediawiki_admin_pass": {
                      "title": "Mediawiki Admin User Password",
                      "type": "string"
                    },
                    "mediawiki_admin_user": {
                      "default": "admin",
                      "title": "Mediawiki Admin User",
                      "type": "string"
                    },
                    "mediawiki_db_schema": {
                      "default": "mediawiki",
                      "title": "Mediawiki DB Schema",
                      "type": "string"
                    },
                    "mediawiki_site_lang": {
                      "default": "en",
                      "title": "Mediawiki Site Language",
                      "type": "string"
                    },
                    "mediawiki_site_name": {
                      "default": "MediaWiki",
                      "title": "Mediawiki Site Name",
                      "type": "string"
                    }
                  },
                  "required": [
                    "mediawiki_db_schema",
                    "mediawiki_site_name",
                    "mediawiki_site_lang",
                    "mediawiki_admin_user",
                    "mediawiki_admin_pass"
                  ],
                  "type": "object"
                }
              },
              "update": {}
            },
            "service_binding": {
              "create": {
                "parameters": {
                  "$schema": "http://json-schema.org/draft-04/schema",
                  "additionalProperties": false,
                  "properties": {
                    "mediawiki_admin_pass": {
                      "title": "Mediawiki Admin User Password",
                      "type": "string"
                    },
                    "mediawiki_admin_user": {
                      "default": "admin",
                      "title": "Mediawiki Admin User",
                      "type": "string"
                    },
                    "mediawiki_db_schema": {
                      "default": "mediawiki",
                      "title": "Mediawiki DB Schema",
                      "type": "string"
                    },
                    "mediawiki_site_lang": {
                      "default": "en",
                      "title": "Mediawiki Site Language",
                      "type": "string"
                    },
                    "mediawiki_site_name": {
                      "default": "MediaWiki",
                      "title": "Mediawiki Site Name",
                      "type": "string"
                    }
                  },
                  "type": "object"
                }
              }
            }
          }
        }
      ]
    }
  ]
}
Comment 1 John Matthews 2017-07-06 20:41:50 UTC 
Aligning to 3.7.0 

Note for 3.6.0 we only intended to support 2 APBs from registry.access.redhat.com, so I don't consider this applicable for 3.6.0, but it's a good feature for us to implement in next release.
Comment 2 DeShuai Ma 2017-07-11 07:01:10 UTC 
Now after enable service-catalog, asb. There will be create a broker named 'ansible-service-broker'. But when get serviceclass there is nothing. 
At least we need trigger bootstrap in openshift-ansible script.

[root@ip-172-18-6-17 ~]# oc get broker
NAME                     KIND
ansible-service-broker   Broker.v1alpha1.servicecatalog.k8s.io
[root@ip-172-18-6-17 ~]# oc get serviceclass
No resources found.
Let custome manually trigger bootstrap after install is not good user experience. How do you think?
Comment 3 John Matthews 2017-07-11 12:00:15 UTC 
DeShuai,

Agreed we need a mechanism so user does not need to trigger the bootstrap manually.

There was some confusion interpreting this BZ, we thought the BZ was asking for Broker to watch the registry and detect new APBs as they are published, that is desired but outside of 3.6 scope.

Agree for 3.6 the Broker needs to read the registry initially and see the APBs which are published.

I'll move this back to 3.6.0 and we will address shortly.
Comment 4 Erik Nelson 2017-07-11 16:19:36 UTC 
https://github.com/openshift/ansible-service-broker/pull/265
Comment 5 DeShuai Ma 2017-07-19 09:45:54 UTC 
I get below error:
[2017-07-19T09:35:58.072Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: 
[2017-07-19T09:35:58.072Z] [WARNING] invalid character '<' looking for beginning of value



[root@host-8-175-72 ~]# oc get configmap broker-config -o yaml
apiVersion: v1
data:
  broker-config: |
    registry:
      name: "rhcc"
      url:  "xxxxxx"  -> use our stage registry
      user: ""
      pass: ""
      org:  "openshift3"
    dao:
      etcd_host: etcd
      etcd_port: 2379
    log:
      logfile: /var/log/ansible-service-broker/asb.log
      stdout: true
      level: "info"
      color: true
    openshift: {}
    broker:
      dev_broker: false
      launch_apb_on_bind: false
      bootstrap_on_startup: true
      recovery: true
      output_request: false
kind: ConfigMap
metadata:
  creationTimestamp: 2017-07-19T06:29:06Z
  labels:
    app: ansible-service-broker
  name: broker-config
  namespace: openshift-ansible-service-broker
  resourceVersion: "13906"
  selfLink: /api/v1/namespaces/openshift-ansible-service-broker/configmaps/broker-config
  uid: 90f52d80-6c4b-11e7-a0c3-fa163ec489fc
[root@host-8-175-72 ~]# oc get po
NAME                    READY     STATUS    RESTARTS   AGE
asb-1222154024-w8p08    1/1       Running   0          1m
etcd-3119139199-jlhlc   1/1       Running   2          6m
[root@host-8-175-72 ~]# oc logs asb-1222154024-w8p08
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-19T09:35:54.807Z] [INFO] == ETCD CX ==
[2017-07-19T09:35:54.807Z] [INFO] EtcdHost: etcd
[2017-07-19T09:35:54.807Z] [INFO] EtcdPort: 2379
[2017-07-19T09:35:54.807Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-19T09:35:54.822Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0]
[2017-07-19T09:35:54.823Z] [INFO] Cluster Client Created
[2017-07-19T09:35:54.869Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-19T09:35:54.869Z] [INFO] == REGISTRY CX == 
[2017-07-19T09:35:54.869Z] [INFO] Name: rhcc
[2017-07-19T09:35:54.869Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-19T09:35:54.876Z] [INFO] Listening for provision messages
[2017-07-19T09:35:55.349Z] [INFO] Initiating Recovery Process
[2017-07-19T09:35:55.352Z] [INFO] No jobs to recover
[2017-07-19T09:35:55.352Z] [NOTICE] 
[2017-07-19T09:35:55.352Z] [INFO] Broker configured to bootstrap on startup
[2017-07-19T09:35:55.352Z] [INFO] Attempting bootstrap...
[2017-07-19T09:35:55.352Z] [INFO] AnsibleBroker::Bootstrap
[2017-07-19T09:35:58.072Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: 
[2017-07-19T09:35:58.072Z] [WARNING] invalid character '<' looking for beginning of value
[2017-07-19T09:35:58.072Z] [NOTICE] Broker successfully bootstrapped on startup
[2017-07-19T09:35:58.072Z] [NOTICE] Ansible Service Broker Started
[2017-07-19T09:35:58.072Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-19T09:36:11.023Z] [INFO] AnsibleBroker::Catalog
10.129.0.1 - - [19/Jul/2017:09:36:11 +0000] "GET /v2/catalog HTTP/1.1" 200 21
[2017-07-19T09:36:25.811Z] [INFO] AnsibleBroker::Catalog
10.129.0.1 - - [19/Jul/2017:09:36:25 +0000] "GET /v2/catalog HTTP/1.1" 200 21
[2017-07-19T09:36:49.291Z] [INFO] AnsibleBroker::Catalog
10.129.0.1 - - [19/Jul/2017:09:36:49 +0000] "GET /v2/catalog HTTP/1.1" 200 21
Comment 9 weiwei jiang 2017-07-20 07:15:10 UTC 
Checked again and found the issue can not be reproduced now, and bootstrap_on_startup work well.

# oc logs -f asb-1222154024-phdgn 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-20T06:04:07.195Z] [INFO] == ETCD CX ==
[2017-07-20T06:04:07.195Z] [INFO] EtcdHost: etcd
[2017-07-20T06:04:07.195Z] [INFO] EtcdPort: 2379
[2017-07-20T06:04:07.195Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-20T06:04:07.216Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0]
[2017-07-20T06:04:07.217Z] [INFO] Cluster Client Created
[2017-07-20T06:04:07.244Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-20T06:04:07.244Z] [INFO] == REGISTRY CX == 
[2017-07-20T06:04:07.244Z] [INFO] Name: rhcc
[2017-07-20T06:04:07.244Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-20T06:04:07.245Z] [INFO] Listening for provision messages
[2017-07-20T06:04:07.732Z] [INFO] Initiating Recovery Process
[2017-07-20T06:04:07.734Z] [INFO] Recovery complete
[2017-07-20T06:04:07.734Z] [NOTICE] recover called
[2017-07-20T06:04:07.734Z] [INFO] Broker configured to bootstrap on startup
[2017-07-20T06:04:07.734Z] [INFO] Attempting bootstrap...
[2017-07-20T06:04:07.734Z] [INFO] AnsibleBroker::Bootstrap
[2017-07-20T06:04:12.125Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping.
[2017-07-20T06:04:12.462Z] [NOTICE] Broker successfully bootstrapped on startup
[2017-07-20T06:04:12.462Z] [NOTICE] Ansible Service Broker Started
[2017-07-20T06:04:12.462Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-20T06:04:24.468Z] [INFO] AnsibleBroker::Catalog


# apb list --broker http://asb-1338-openshift-ansible-service-broker.0719-92s.qe.rhcloud.com     
ID                                    NAME            DESCRIPTION                        
e1bcd4a8-ce02-4585-9f4c-18abd56d76f2  postgresql-apb  SCL PostgreSQL apb implementation  
4fbcc051-03af-40c7-86e2-dc12977e6b5d  mediawiki-apb   Mediawiki123 apb implementation
Comment 10 John Matthews 2017-07-24 18:41:08 UTC 
QE Please re-test, let us know if you require anything further to close this issue out.

The issue seen in comment #5 we think is due to an error from stage registry when it returned html data instead of json content.

Perhaps it's related to configuring with https vs http for the stage repo.
Comment 11 John Matthews 2017-07-24 19:42:31 UTC 
The issue is related to bad response from stage registry. 





Configured with http for stage
ansible_service_broker_registry_url: "http://registry.access.stage.redhat.com"


Saw error:
[2017-07-24T19:08:56.241Z] [INFO] Broker configured to bootstrap on startupESC[0m
[2017-07-24T19:08:56.241Z] [INFO] Attempting bootstrap...ESC[0m
[2017-07-24T19:08:56.241Z] [INFO] AnsibleBroker::BootstrapESC[0m
ESC[36m[2017-07-24T19:08:56.241Z] [DEBUG] Dao::BatchGetRawESC[0m
ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] RHCCRegistry::LoadSpecsESC[0m
ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] RHCCRegistry::LoadImagesESC[0m
ESC[36m[2017-07-24T19:08:56.242Z] [DEBUG] Using http://registry.access.stage.redhat.com to source APB images using query:"*-apb"ESC[0m
ESC[36m[2017-07-24T19:08:56.669Z] [DEBUG] Got Image Response from RHCCESC[0m
ESC[33m[2017-07-24T19:08:56.669Z] [WARNING] Failed to bootstrap on startup! You may need to manually trigger a bootstrap to resync APB inventory. See error: ESC[0m
ESC[33m[2017-07-24T19:08:56.669Z] [WARNING] invalid character '<' looking for beginning of valueESC[0m
ESC[32m[2017-07-24T19:08:56.669Z] [NOTICE] Broker successfully bootstrapped on startupESC[0m
ESC[32m[2017-07-24T19:08:56.669Z] [NOTICE] Ansible Service Broker StartedESC[0m


~30 minutes later manually triggered bootstrap and no issues.
I did not make any other changes.

# curl -X POST http://asb-1338-openshift-ansible-service-broker.apps.192.168.120.4.nip.io/v2/bootstrap
{
  "spec_count": 2,
  "image_count": 3
}



[2017-07-24T19:39:04.239Z] [INFO] AnsibleBroker::Bootstrap
[2017-07-24T19:39:04.239Z] [DEBUG] Dao::BatchGetRaw
[2017-07-24T19:39:04.24Z] [DEBUG] RHCCRegistry::LoadSpecs
[2017-07-24T19:39:04.24Z] [DEBUG] RHCCRegistry::LoadImages
[2017-07-24T19:39:04.24Z] [DEBUG] Using http://registry.access.stage.redhat.com to source APB images using query:"*-apb"
[2017-07-24T19:39:06.613Z] [DEBUG] Got Image Response from RHCC
[2017-07-24T19:39:06.613Z] [DEBUG] Properly unmarshalled image response
[2017-07-24T19:39:06.614Z] [DEBUG] Found 3 images in RHCC
[2017-07-24T19:39:06.614Z] [DEBUG] RHCCRegistry::imageToSpec
[2017-07-24T19:39:06.953Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping.
[2017-07-24T19:39:06.953Z] [DEBUG] RHCCRegistry::imageToSpec
[2017-07-24T19:39:07.149Z] [DEBUG] Successfully converted RHCC Image postgresql-apb into Spec
[2017-07-24T19:39:07.149Z] [DEBUG] RHCCRegistry::imageToSpec
[2017-07-24T19:39:07.342Z] [DEBUG] Successfully converted RHCC Image mediawiki-apb into Spec
10.128.0.1 - - [24/Jul/2017:19:39:04 +0000] "POST /v2/bootstrap HTTP/1.1" 200 42
Comment 12 John Matthews 2017-07-24 20:27:22 UTC 
Our understanding of this issue is that sometimes the stage registry will return a bad response.

When this happens, the Broker logs a warning that auto bootstrap fails and broker continues.

User is left with a service catalog lacking APBs.  The cause is not obvious, User needs to look at ASB logs and search for WARNING entries to see that auto bootstrap failed.

Next they need to trigger a manual bootstrap and wait for Service Catalog relist the Broker, then APBs are available in WebUI.

To improve this experience we will make a change for 3.6.0 that if auto bootstrap fails the broker will fail loudly and stop.  Kube will retry the pod, eventually if the response from the registry is good the bootstrap will succeed.
Comment 13 Fabian von Feilitzsch 2017-07-24 21:11:18 UTC 
Made the broker fail on error.

https://github.com/openshift/ansible-service-broker/pull/300
Comment 16 weiwei jiang 2017-07-31 05:59:13 UTC 
Checked with ansible-service-broker 0.9.11, and found the bootstrap_on_startup work well, and the pod will stop and kube will retry if bootstrap failed. so verify this issue.