Bug 1572449

Summary: Stage registry is returning "is_official" and "is_trusted" as String value, while production returns them as Boolean
Product: OpenShift Container Platform Reporter: Zihan Tang <zitang>
Component: Service BrokerAssignee: John Matthews <jmatthew>
Status: CLOSED CURRENTRELEASE QA Contact: Zihan Tang <zitang>
Severity: medium Docs Contact:
Priority: high    
Version: 3.10.0CC: aos-bugs, jiazha, jmatthew, peasters, zhsun
Target Milestone: ---Keywords: TestBlocker
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
This was an issue with an internal service, fix was to configure the stage service to mimic production.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-20 21:12:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1565241, 1569220, 1569708, 1570106, 1570107, 1570108    

Description Zihan Tang 2018-04-27 03:24:02 UTC
Description of problem:
asb is unable to retrieve image names for stage registry

Version-Release number of selected component (if applicable):
asb : 1.2.7

How reproducible:
always

Steps to Reproduce:
1.set broker-config, using stage registry
 registry:
  - type: rhcc
    name: rh
    url:  registry.access.stage.redhat.com
    org:  
    tag:  v3.10
    white_list: [.*-apb$]
2. check asb logs and apb bundles

Actual results:
asb can not retrieve image names for stage registry
asb log:
time="2018-04-27T03:15:08Z" level=error msg="unable to retrieve image names for registry rh - json: cannot unmarshal string into Go struct field RHCCImage.is_official of type bool"
[2018-04-27T03:15:08.817Z] [WARNING] - registry: 0x157f290 was unable to complete bootstrap - json: cannot unmarshal string into Go struct field RHCCImage.is_official of type bool

[root@host-172-16-120-175 ~]# oc get bundles
No resources found.

Expected results:
retrieve image names and spec successfully.

Additional info:
asb can retrieve images from dockerhub and access registry.

Comment 1 Shawn Hurley 2018-04-27 17:47:37 UTC
Stage request output: 

* Notice the "is_official" and "is_trusted" types

{"num_results": 7, "query": "\"*-apb\"", "results": [{"description": null, "is_official": "false", "is_trusted": "false", "name": "MySQL APB", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "MariaDB APB", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "MediaWiki APB", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "Ansible Playbook Bundle Base", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "PostgreSQL APB", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "Ansible Playbook Bundle (APB) Tools", "should_filter": false, "star_count": 0}, {"description": null, "is_official": "false", "is_trusted": "false", "name": "Ansible Service Broker", "should_filter": false, "star_count": 0}]}


Production request output:

* Notice the "is_official" and "is_trusted" types

{"num_results": 6, "query": "\"*-apb\"", "results": [{"description": "Ansible Playbook Bundle (APB) tools to assist in the building and packaging of APBs.", "is_official": false, "is_trusted": false, "name": "openshift3/apb-tools", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MySQL.MySQL is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mysql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundles Base Image, contains an Ansible runtime along with Kubernetes and OpenShift modules", "is_official": false, "is_trusted": false, "name": "openshift3/apb-base", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MariaDB.MariaDB is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mariadb-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for PostgreSQL. PostgreSQL is an advanced Object-Relational database management system (DBMS)", "is_official": false, "is_trusted": false, "name": "openshift3/postgresql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MediaWiki. MediaWiki is a free software open source wiki package written in PHP, originally for use on Wikipedia. It is now also used by several other projects of the non-profit Wikimedia Foundation and by many other wikis", "is_official": false, "is_trusted": false, "name": "openshift3/mediawiki-apb", "should_filter": true, "star_count": 0}]}

Comment 2 John Matthews 2018-04-27 18:54:40 UTC
The underlying problem looks to be that the stage and production environment have different behavior.

The fix needed is to update the stage environment so it has the same behavior.

Comment 3 Patrick Easters 2018-04-30 00:21:46 UTC
There was a change intended for validation in QA that was accidentally merged into stage as well. This was reverted Friday and the stage registry should now return the expected types for those fields.

Comment 4 Zihan Tang 2018-05-02 03:19:19 UTC
John & Patrick, 
the stage registry is still not reachable . 
in asb 1.2.7 , still report the log: 

time="2018-05-02T03:13:59Z" level=error msg="unable to retrieve image names for registry rh - 500 INTERNAL SERVER ERROR"
[2018-05-02T03:13:59.862Z] [WARNING] - registry: 0x157f290 was unable to complete bootstrap - 500 INTERNAL SERVER ERROR

Comment 5 John Matthews 2018-05-02 16:59:46 UTC
Putting this back ON_QA, to ask for re-test. 

I tried the stage registry today and looks OK now.


$ curl -i -k  -H "Accept: application/json" "https://registry.access.stage.redhat.com/v1/search?q=apb"
HTTP/1.1 200 OK
Date: Wed, 02 May 2018 16:55:09 GMT
Server: Apache
Content-Length: 1712
X-Docker-Registry-Version: 0.6.6
X-Docker-Registry-Config: common
Connection: close
Content-Type: application/json

{"num_results": 6, "query": "apb", "results": [{"description": "Ansible Playbook Bundle (APB) tools to assist in the building and packaging of APBs.", "is_official": false, "is_trusted": false, "name": "openshift3/apb-tools", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MariaDB.MariaDB is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mariadb-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MySQL.MySQL is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mysql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundles Base Image, contains an Ansible runtime along with Kubernetes and OpenShift modules", "is_official": false, "is_trusted": false, "name": "openshift3/apb-base", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for PostgreSQL. PostgreSQL is an advanced Object-Relational database management system (DBMS)", "is_official": false, "is_trusted": false, "name": "openshift3/postgresql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MediaWiki. MediaWiki is a free software open source wiki package written in PHP, originally for use on Wikipedia. It is now also used by several other projects of the non-profit Wikimedia Foundation and by many other wikis", "is_official": false, "is_trusted": false, "name": "openshift3/mediawiki-apb", "should_filter": true, "star_count": 0}]}

Comment 6 Zihan Tang 2018-05-03 02:59:01 UTC
At present , stage service is unavailable.
[root@host-172-16-120-48 ~]#  curl -i -k  -H "Accept: application/json" "https://registry.access.stage.redhat.com/v1/search?q=apb"
HTTP/1.1 503 SERVICE UNAVAILABLE
Date: Thu, 03 May 2018 02:57:04 GMT
Server: Apache
Content-Length: 19
X-Docker-Registry-Version: 0.6.6
X-Docker-Registry-Config: common
Connection: close
Content-Type: text/html; charset=utf-8

Service Unavailable

And pull image failed:
 docker pull registry.access.stage.redhat.com/openshift3/mariadb-apb:v3.9
Trying to pull repository registry.access.stage.redhat.com/openshift3/mariadb-apb ... 
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/mariadb-apb/manifests/list/v3.9 was not found on this server.</p>\n<hr>\n<address>Apache Server at cdn.stage.redhat.com Port 80</address>\n</body></html>\n"

Comment 7 Zihan Tang 2018-05-03 09:53:34 UTC
Now it is available. marked it as verified.
[root@dhcp-140-42 ~]# curl -i -k  -H "Accept: application/json" "https://registry.access.stage.redhat.com/v1/search?q=apb"
HTTP/1.1 200 OK
Date: Thu, 03 May 2018 09:48:19 GMT
Server: Apache
Content-Length: 1712
X-Docker-Registry-Version: 0.6.6
X-Docker-Registry-Config: common
Connection: close
Content-Type: application/json

{"num_results": 6, "query": "apb", "results": [{"description": "Ansible Playbook Bundle (APB) tools to assist in the building and packaging of APBs.", "is_official": false, "is_trusted": false, "name": "openshift3/apb-tools", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MariaDB.MariaDB is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mariadb-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MySQL.MySQL is a multi-user, multi-threaded SQL database server.", "is_official": false, "is_trusted": false, "name": "openshift3/mysql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundles Base Image, contains an Ansible runtime along with Kubernetes and OpenShift modules", "is_official": false, "is_trusted": false, "name": "openshift3/apb-base", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for PostgreSQL. PostgreSQL is an advanced Object-Relational database management system (DBMS)", "is_official": false, "is_trusted": false, "name": "openshift3/postgresql-apb", "should_filter": true, "star_count": 0}, {"description": "Ansible Playbook Bundle application definition for MediaWiki. MediaWiki is a free software open source wiki package written in PHP, originally for use on Wikipedia. It is now also used by several other projects of the non-profit Wikimedia Foundation and by many other wikis", "is_official": false, "is_trusted": false, "name": "openshift3/mediawiki-apb", "should_filter": true, "star_count": 0}]}