Description of problem:

Image pull through fails when we refer registry-mirror in the imagestream for containerized OCP install on OpenStack

Please refer attachment ocp_registry_image_pullthrough_failure_symptoms.txt  for more details

Version-Release number of selected component (if applicable):
containerized install of OCP 3.9 on RHEL

How reproducible:
Always on customer side

Steps to Reproduce:
1.  oc new-project test2
2.   oc project test2


3. Create secret from docker config json file. 
  oc secrets new-dockercfg secret --docker-server=<your mirror>  --docker-username=user  --docker-password="pass" --docker-email=email

secret/secret

4. oc secrets link builder secret --for=pull oc secrets link default 
 secret --for=pull oc secrets link deployer secret --for=pull
 

But meanwhile, we have switched to specifying the secret as JSON file, i.e.

$ cat docker-config.json
{
        "auths": {
                "hostname.example.com:4000": {
                        "auth": "aaaaaaaaaaaaaaaaaaaa"
                },
                "x.y.z.r:4000": {
                        "auth": "aaaaaaaaaaaaaaaaaaaa"
                }
		}	
}


$ oc secrets hostname.example.com docker-config.json && for sa in default deployer builder; do oc secret link $sa hostname.example.com --for=pull; done

5.
We create the image stream with "oc create -f... ", essentially by copying it from the "openshift" namespace -- please see attached YAML spec rubyis.yml for details.

Check attached ocp_registry_image_pullthrough_failure_symptoms.txt for more details.

This does not look like the right way to enable image pullthrough.  You've just created an imagestreamtag that points to the internal registry, when such an image does not exist in the internal registry.

Here are the docs on how to enable pullthrough for an imagestreamtag:

https://docs.openshift.org/latest/dev_guide/managing_images.html#adding-tag

https://docs.openshift.org/latest/install_config/registry/extended_registry_configuration.html#middleware-repository-pullthrough


"If you want to instruct Docker to always fetch the tagged image from the integrated registry, use --reference-policy=local. The registry uses the pull-through feature pull-through feature to serve the image to the client. By default, the image blobs are mirrored locally by the registry. As a result, they can be pulled more quickly the next time they are needed. The flag also allows for pulling from insecure registries without a need to supply --insecure-registry to the Docker daemon as long as the image stream has an insecure annotation or the tag has an insecure import policy."

In other words, you should be taking an imagestreamtag that points to an external image, and updating it to use a local reference policy.  Then when you reference that imagestream, you'll get pullthrough behavior.


Oleg, please double check my assessment in case i have missed something about what the customer has done.

You need to provide the docker secret that can access the external image/repository in the namespace containing the imagestream.

That secret will be used to import the metadata.  You do not need the secret in the namespace that is consuming the image.

I see at least two potential issues:


1) your secrets are of type "opaque" meaning I do not think they were created properly.  (they should be type docker).  See: 
https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries

for instructions on how to create proper docker secrets.  Specifically I would recommend this syntax, given that you have a .docker/config.json format:

$ oc create secret generic <pull_secret_name> \
    --from-file=.dockerconfigjson=<path/to/.docker/config.json> \
    --type=kubernetes.io/dockerconfigjson

I do not believe the secret needs to be linked to anything for it to be used for to work.

2) your imagestream is marked w/ reference policy source, so they are not enabled for pullthrough.  That's not causing the import errors, but it's going to cause you problems once you get through the import error

Miheer, please let us know the result of my suggestions above.

Miheer, bump.

(In reply to Ben Parees from comment #13)
> Miheer, bump.

creating the secret with explicit type docker solved the registry authorization problem for the image stream.

Great, so can this be closed?