Bug 1509366

Summary: Error syncing catalog from ServiceBroker
Product: OpenShift Container Platform Reporter: Qixuan Wang <qixuan.wang>
Component: Service BrokerAssignee: Jesus M. Rodriguez <jesusr>
Status: CLOSED ERRATA QA Contact: Qixuan Wang <qixuan.wang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.7.0CC: aos-bugs, chezhang, jesusr, jmatthew, xtian
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-18 13:23:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1507617    
Bug Blocks:    

Description Qixuan Wang 2017-11-03 15:56:56 UTC 
Description of problem:
Setup service catalog and ansible service broker cluster with ansible installer. Error syncing catalog from ServiceBroker. Error getting catalog payload for broker "ansible-service-broker"; received zero services; at least one service is required. The log shows "[ERROR] V1 Schema Manifest does not exist in registry".

Version-Release number of selected component (if applicable):
openshift v3.7.0-0.191.0
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.7.0-0.191.0
registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker:v3.7.0-0.191.0

How reproducible:
Always

Steps to Reproduce:
1. Setup service catalog and ansible service broker cluster with ansible installer.
2. Check service catalog.
3. Check ansible service broker.


Actual results:
2. [root@host-172-16-120-84 ~]# oc get all -n kube-service-catalog 
NAME               HOST/PORT                                                     PATH      SERVICES    PORT      TERMINATION   WILDCARD
routes/apiserver   apiserver-kube-service-catalog.apps.1103-k5l.qe.rhcloud.com             apiserver   secure    passthrough   None

NAME                          READY     STATUS    RESTARTS   AGE
po/apiserver-5jlrp            1/1       Running   0          2m
po/controller-manager-cc2zm   1/1       Running   1          2m

NAME                     CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
svc/apiserver            172.30.22.88    <none>        443/TCP    2m
svc/controller-manager   172.30.224.19   <none>        6443/TCP   2m

3. [root@host-172-16-120-84 ~]# oc get all -n openshift-ansible-service-broker 
NAME                    REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfigs/asb   1          1         1         config

NAME              HOST/PORT                                                                PATH      SERVICES   PORT      TERMINATION   WILDCARD
routes/asb-1338   asb-1338-openshift-ansible-service-broker.apps.1103-k5l.qe.rhcloud.com             asb        1338      reencrypt     None

NAME             READY     STATUS    RESTARTS   AGE
po/asb-1-tcczv   2/2       Running   1          3m

NAME       DESIRED   CURRENT   READY     AGE
rc/asb-1   1         1         1         3m

NAME      CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
svc/asb   172.30.19.150   <none>        1338/TCP   3m

[root@host-172-16-120-84 ~]# oc logs po/asb-1-tcczv -c asb -n openshift-ansible-service-broker 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-11-03T15:38:04.8Z] [NOTICE] Initializing clients...
[2017-11-03T15:38:04.8Z] [INFO] == ETCD CX ==
[2017-11-03T15:38:04.8Z] [INFO] EtcdHost: 0.0.0.0
[2017-11-03T15:38:04.8Z] [INFO] EtcdPort: 2379
[2017-11-03T15:38:04.8Z] [INFO] Endpoints: [http://0.0.0.0:2379]
[2017-11-03T15:38:05.765Z] [INFO] Etcd Version [Server: 3.2.7, Cluster: not_decided]
[2017-11-03T15:38:05.776Z] [INFO] Kubernetes version: v1.7.6+a08f5eeb62
[2017-11-03T15:38:05.777Z] [INFO] == REGISTRY CX == 
[2017-11-03T15:38:05.777Z] [INFO] Name: rh
[2017-11-03T15:38:05.777Z] [INFO] Type: dockerhub
[2017-11-03T15:38:05.777Z] [INFO] Url: https://registry.hub.docker.com
[2017-11-03T15:38:05.777Z] [INFO] No cluster credential overrides provided, using k8s InClusterConfig
[2017-11-03T15:38:05.777Z] [INFO] Listening for provision messages
[2017-11-03T15:38:05.777Z] [INFO] Listening for deprovision messages
[2017-11-03T15:38:05.777Z] [INFO] Listening for provision messages
[2017-11-03T15:38:06.062Z] [INFO] Initiating Recovery Process
[2017-11-03T15:38:06.063Z] [INFO] No jobs to recover
[2017-11-03T15:38:06.063Z] [NOTICE] 
[2017-11-03T15:38:06.063Z] [INFO] Broker configured to bootstrap on startup
[2017-11-03T15:38:06.063Z] [INFO] Attempting bootstrap...
[2017-11-03T15:38:06.063Z] [INFO] AnsibleBroker::Bootstrap
[2017-11-03T15:38:07.109Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.157Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.215Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.259Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.311Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.391Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.443Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.567Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.621Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.714Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.764Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.813Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.858Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.908Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:07.958Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:08.007Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:08.057Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:08.107Z] [ERROR] V1 Schema Manifest does not exist in registry
[2017-11-03T15:38:08.107Z] [INFO] Validating specs...
[2017-11-03T15:38:08.107Z] [NOTICE] All specs passed validation!
[2017-11-03T15:38:08.107Z] [NOTICE] Broker successfully bootstrapped on startup
[2017-11-03T15:38:08.278Z] [NOTICE] Listening on https://0.0.0.0:1338
[2017-11-03T15:38:08.278Z] [NOTICE] Ansible Service Broker Starting
[2017-11-03T15:38:58.55Z] [INFO] AnsibleBroker::Catalog
10.128.0.1 - - [03/Nov/2017:15:38:58 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 21
[2017-11-03T15:38:58.579Z] [INFO] AnsibleBroker::Catalog
10.128.0.1 - - [03/Nov/2017:15:38:58 +0000] "GET /ansible-service-broker/v2/catalog HTTP/1.1" 200 21
[2017-11-03T15:38:58.773Z] [INFO] AnsibleBroker::Catalog


[root@host-172-16-120-84 ~]# oc describe clusterservicebroker ansible-service-broker 
Name:		ansible-service-broker
Namespace:	
Labels:		<none>
Annotations:	<none>
API Version:	servicecatalog.k8s.io/v1beta1
Kind:		ClusterServiceBroker
Metadata:
  Creation Timestamp:	2017-11-03T15:37:15Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generation:		1
  Resource Version:	3461
  Self Link:		/apis/servicecatalog.k8s.io/v1beta1/clusterservicebrokers/ansible-service-broker
  UID:			de0b11a9-c0ac-11e7-856d-0a580a800003
Spec:
  Auth Info:
    Bearer:
      Secret Ref:
        Name:		asb-client
        Namespace:	openshift-ansible-service-broker
  Ca Bundle:		xxx
  Relist Behavior:	Duration
  Relist Duration:	15m0s
  Relist Requests:	0
  URL:			https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker
Status:
  Conditions:
    Last Transition Time:	2017-11-03T15:37:18Z
    Message:			Error syncing catalog from ServiceBroker. Error getting catalog payload for broker "ansible-service-broker"; received zero services; at least one service is required
    Reason:			ErrorSyncingCatalog
    Status:			False
    Type:			Ready
  Reconciled Generation:	0
Events:
  FirstSeen	LastSeen	Count	From					SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----					-------------	--------	------			-------
  2m		1m		14	service-catalog-controller-manager			Warning		ErrorFetchingCatalog	Error getting broker catalog: Get https://asb.openshift-ansible-service-broker.svc:1338/ansible-service-broker/v2/catalog: dial tcp 172.30.19.150:1338: getsockopt: no route to host
  27s		25s		11	service-catalog-controller-manager			Warning		ErrorSyncingCatalog	Error getting catalog payload for broker "ansible-service-broker"; received zero services; at least one service is required


Expected results:
3. Ansible service broker should run successfully.

Additional info:
Comment 1 John Matthews 2017-11-06 13:31:28 UTC 
This looks like an error related to bad credentials for the registry adapter.
Comment 2 Jesus M. Rodriguez 2017-11-06 15:31:18 UTC 
The changes to the installer have not yet been merged:

https://github.com/openshift/openshift-ansible/pull/6009
Comment 3 Jesus M. Rodriguez 2017-11-06 15:39:35 UTC 
As John mentioned in comment #1, this is a dockerhub credentials issue. Ensure that the username and password used for the ansible_service_broker_registry_user and ansible_service_broker_registry_password entries are valid credentials.

This should allow the broker to retrieve the catalog metadata and the service-catalog to again receive the list of services.
Comment 4 Zhang Cheng 2017-11-08 07:00:21 UTC 
ansible-service-broker deploy failed while using openshift-ansible with latest build openshift-ansible-3.7.0-0.197.0

Refer to bug: https://bugzilla.redhat.com/show_bug.cgi?id=1507617
Comment 5 Zhang Cheng 2017-11-09 02:43:56 UTC 
Retested and verified with openshift-ansible-3.7.4-1
svc-catalog and asb can be deployed succeed.
Comment 8 errata-xmlrpc 2017-12-18 13:23:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3464