Related downstream PR: https://github.com/openshift/openshift-ansible/pull/6009

Dylan, Could you help to provide the detail steps to verify this bug? Thanks.

Hey Zhang,

All of your test steps are great but there are couple of things you misconfigured. To use this new adapter you need to have type: local_openshift and define which namespaces you want the broker to load APBs from. This is what a valid registry config looks like:

    registry:
      - type: local_openshift
        name: lo
        tag:  latest
        namespaces:
          - openshift
        white_list: [u'.*-apb$']

In this instance the 'namespace' parameter aligns with the organization in the tag. So you need to tag the docker image: 172.30.187.214:5000/openshift/rhscl-postgresql-apb:latest and push it to the registry. The openshift namespace is already populated with sample images so it's standard practice to use this namespace to host our APBs.

Please let me know if you run into any more trouble testing this. I am moving back to ON_QA because the last test tested the wrong registry adapter.

Please ignore the comment 8, that is a mistake

Let me summary the current problems and hope confirm with you.
1. I didn't find any description about how to set local_openshift type registry, do we need a bug to trace doc?

2. The original problem in bug description verify failed since pr https://github.com/openshift/openshift-ansible/pull/6009 is not merged.

3. Cannot get any service after bootstrap since "Image does not have proper registry IP prefix. Something went wrong." while using workaround. Please let me know if you want to fix in here or open a new bug.

Zhang,

1. I submitted some documentation for using this registry adapter here:

https://github.com/openshift/ansible-service-broker/pull/538
https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/pull/161

2. Yes until that PR is merged the service account downstream won't be able to access the imagestream resource. Your workaround is sufficient.

3. It appears that the image you pushed was not tagged with the IP address of the registry service but rather the hostname. Was that intentional? I'm curious if that happened automatically. I will put a fix in to not error out when a hostname is present.

3. As I said in comment 10, I used ip:pory to push images，so I think it maybe generaged by automatically.
Push image to local openshift images
[root@host-172-16-120-117 ~]# docker push 172.30.42.85:5000/ose-registry/rhscl-postgresql-apb:latest
[root@host-172-16-120-117 ~]# docker push 172.30.42.85:5000/ose-registry/mediawiki123-apb:latest

Thanks Zhang, I have fixed this problem here: https://github.com/openshift/ansible-service-broker/pull/540.

I changed the error to a warning statement and proceed to add the APB if it's in the configured namespace.

Although PR https://github.com/openshift/openshift-ansible/pull/6009 was merged. But ansible-service-broker deploy failed while using openshift-ansible with latest build openshift-ansible-3.7.0-0.197.0

Refer to bug: https://bugzilla.redhat.com/show_bug.cgi?id=1507617

That means APBs of local_openshift registry can be listed, but provision failed and serviceinstance cannot be deleted either.

Please review if it is a block issue for GA. Thanks.

Unable to reproduce upstream. Somehow the registry IP is being converted to the service hostname in a downstream environment. I am bringing up a full downstream test environment to see if I can reproduce.

Unable to reproduce downstream as well. I confirmed using the same test steps that the IP address was converted into a hostname but I did not see an error when provisioning. This error sounds environment related where the node that the pods are being deployed too cannot resolve the service hostname. 

I plan on opening a new bug to track this error but I do not believe there is a bug with the registry adapter.

Actually I guess this is a authorization issue. Means user in transient namespace have no permission to pull images from local_openshift registry.

Weiwei and Zhang,

I need to know which user you are creating the new project with which you push the images to. I.e. in the most recent test: namespace `test-ip`. I believe Weiwei is correct that it is an authorization issue because in my tests I am using the preconfigured `openshift` namespace. 

I noticed on your test environment that project `test-ip` was created and owned by user `system:admin`. You then push the images to the registry under namespace `test-ip`. I then have to assume you login to the web console as someone OTHER than `system:admin` because the web console requires a user/pass combo. When you are then logged in as someone else (I'm assuming user `chezhang` because that was the only other user I could find using the namespaces) and try to deploy the image the transient namespace only has access to imagestreams in the namespaces he has created.

By default we configure the broker to look at the 'openshift' namespace since these imagestreams are available to cluster users. The 'openshift' namespace is a shared resource namespace so all authenticated users have access to the imagestreams in this namespace. If you want to use a different namespace then it has to have elevated permissions to expose its imagestreams to all authenticated users.

If you repeat all of these steps but instead use the 'openshift' namespace I am confident you will not see the ErrImgPull errors.

Yes. I used chezhang user to create project test-ip and push images. I understand your explainatiom，But, local openshift registry work well and provision succeed while using domain name by same steps(not use openshift namespace, create project and push image by chezhang user), it is confused for user.
Furthermore, I think we should doc the limitation about user and namespace of local registry if you think that is expect result. Thanks.

Zhang,

It shouldn't matter what you tag and push the image as. I tested on your machine even with tagging the images using the domain name and I still saw the same errors. I believe it will solely depend on which user creates the namespace where the imagestreams live and whether the apb sandbox that gets created will have access to the imagestreams in that namespace.

I don't believe the local registry will act different simply based on whether the image is tagged with the IP vs domain name. I was unable to reproduce a successful provisioning unless I gave all authenticated users access to the imagestreams resource in that namespace.

I agree that documenting the proper way for an administrator to configure this adapter makes sense. I can add comments here: https://bugzilla.redhat.com/show_bug.cgi?id=1511656 to include explanation around working with the "openshift" namespace and using others.

Please retest using the openshift namespace and/or using an namespace that exposes imagestreams to all authenticated users.

Zhang,

https://docs.openshift.com/enterprise/3.2/dev_guide/managing_images.html#allowing-pods-to-reference-images-across-projects

The above documentation explains how to enable users to access images from other namespaces. It might be useful if you choose to expose imagestreams from a test namespace.

Zhang,

I have opened this bug (https://bugzilla.redhat.com/show_bug.cgi?id=1512042) for next release to cover the problems you have encountered. I have also updated the docs bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1511656 to encapsulate what issues users might expect if they don't use the 'openshift' namespace.

Dylan,

Retested and LGTM, I'm changing status to "Verified". Thanks for you opened two bugs for limitation and doc. I will continue to trace them.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188