Bug 1571079 - Image pull through fails when we refer registry-mirror in the imagestream for containerized OCP install on OpenStack
Summary: Image pull through fails when we refer registry-mirror in the imagestream for...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.0
Assignee: Ben Parees
QA Contact: Dongbo Yan
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-24 05:11 UTC by Miheer Salunke
Modified: 2018-05-07 19:31 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-05-07 19:31:12 UTC
Target Upstream Version:

Attachments (Terms of Use)

Comment 1 Miheer Salunke 2018-04-24 05:18:29 UTC
Description of problem:

Image pull through fails when we refer registry-mirror in the imagestream for containerized OCP install on OpenStack

Please refer attachment ocp_registry_image_pullthrough_failure_symptoms.txt  for more details

Version-Release number of selected component (if applicable):
containerized install of OCP 3.9 on RHEL

How reproducible:
Always on customer side

Steps to Reproduce:
1.  oc new-project test2
2.   oc project test2

3. Create secret from docker config json file. 
  oc secrets new-dockercfg secret --docker-server=<your mirror>  --docker-username=user  --docker-password="pass" --docker-email=email@gmail.com


4. oc secrets link builder secret --for=pull oc secrets link default 
 secret --for=pull oc secrets link deployer secret --for=pull

But meanwhile, we have switched to specifying the secret as JSON file, i.e.

$ cat docker-config.json
        "auths": {
                "hostname.example.com:4000": {
                        "auth": "aaaaaaaaaaaaaaaaaaaa"
                "x.y.z.r:4000": {
                        "auth": "aaaaaaaaaaaaaaaaaaaa"

$ oc secrets hostname.example.com docker-config.json && for sa in default deployer builder; do oc secret link $sa hostname.example.com --for=pull; done

We create the image stream with "oc create -f... ", essentially by copying it from the "openshift" namespace -- please see attached YAML spec rubyis.yml for details.

Check attached ocp_registry_image_pullthrough_failure_symptoms.txt for more details.

Comment 5 Ben Parees 2018-04-24 13:10:04 UTC
This does not look like the right way to enable image pullthrough.  You've just created an imagestreamtag that points to the internal registry, when such an image does not exist in the internal registry.

Here are the docs on how to enable pullthrough for an imagestreamtag:



"If you want to instruct Docker to always fetch the tagged image from the integrated registry, use --reference-policy=local. The registry uses the pull-through feature pull-through feature to serve the image to the client. By default, the image blobs are mirrored locally by the registry. As a result, they can be pulled more quickly the next time they are needed. The flag also allows for pulling from insecure registries without a need to supply --insecure-registry to the Docker daemon as long as the image stream has an insecure annotation or the tag has an insecure import policy."

In other words, you should be taking an imagestreamtag that points to an external image, and updating it to use a local reference policy.  Then when you reference that imagestream, you'll get pullthrough behavior.

Oleg, please double check my assessment in case i have missed something about what the customer has done.

Comment 7 Ben Parees 2018-04-24 13:29:17 UTC
You need to provide the docker secret that can access the external image/repository in the namespace containing the imagestream.

That secret will be used to import the metadata.  You do not need the secret in the namespace that is consuming the image.

Comment 11 Ben Parees 2018-04-27 03:09:03 UTC
I see at least two potential issues:

1) your secrets are of type "opaque" meaning I do not think they were created properly.  (they should be type docker).  See: 

for instructions on how to create proper docker secrets.  Specifically I would recommend this syntax, given that you have a .docker/config.json format:

$ oc create secret generic <pull_secret_name> \
    --from-file=.dockerconfigjson=<path/to/.docker/config.json> \

I do not believe the secret needs to be linked to anything for it to be used for to work.

2) your imagestream is marked w/ reference policy source, so they are not enabled for pullthrough.  That's not causing the import errors, but it's going to cause you problems once you get through the import error

Comment 12 Ben Parees 2018-04-27 17:42:50 UTC
Miheer, please let us know the result of my suggestions above.

Comment 13 Ben Parees 2018-05-02 16:17:13 UTC
Miheer, bump.

Comment 14 Ryan Howe 2018-05-03 18:13:43 UTC
(In reply to Ben Parees from comment #13)
> Miheer, bump.

creating the secret with explicit type docker solved the registry authorization problem for the image stream.

Comment 15 Ben Parees 2018-05-03 20:56:57 UTC
Great, so can this be closed?

Note You need to log in before you can comment on or make changes to this bug.