Description of problem:
Image pull through fails when we refer registry-mirror in the imagestream for containerized OCP install on OpenStack
Please refer attachment ocp_registry_image_pullthrough_failure_symptoms.txt for more details
Version-Release number of selected component (if applicable):
containerized install of OCP 3.9 on RHEL
Always on customer side
Steps to Reproduce:
1. oc new-project test2
2. oc project test2
3. Create secret from docker config json file.
oc secrets new-dockercfg secret --docker-server=<your mirror> --docker-username=user --docker-password="pass" --email@example.com
4. oc secrets link builder secret --for=pull oc secrets link default
secret --for=pull oc secrets link deployer secret --for=pull
But meanwhile, we have switched to specifying the secret as JSON file, i.e.
$ cat docker-config.json
$ oc secrets hostname.example.com docker-config.json && for sa in default deployer builder; do oc secret link $sa hostname.example.com --for=pull; done
We create the image stream with "oc create -f... ", essentially by copying it from the "openshift" namespace -- please see attached YAML spec rubyis.yml for details.
Check attached ocp_registry_image_pullthrough_failure_symptoms.txt for more details.
This does not look like the right way to enable image pullthrough. You've just created an imagestreamtag that points to the internal registry, when such an image does not exist in the internal registry.
Here are the docs on how to enable pullthrough for an imagestreamtag:
"If you want to instruct Docker to always fetch the tagged image from the integrated registry, use --reference-policy=local. The registry uses the pull-through feature pull-through feature to serve the image to the client. By default, the image blobs are mirrored locally by the registry. As a result, they can be pulled more quickly the next time they are needed. The flag also allows for pulling from insecure registries without a need to supply --insecure-registry to the Docker daemon as long as the image stream has an insecure annotation or the tag has an insecure import policy."
In other words, you should be taking an imagestreamtag that points to an external image, and updating it to use a local reference policy. Then when you reference that imagestream, you'll get pullthrough behavior.
Oleg, please double check my assessment in case i have missed something about what the customer has done.
You need to provide the docker secret that can access the external image/repository in the namespace containing the imagestream.
That secret will be used to import the metadata. You do not need the secret in the namespace that is consuming the image.
I see at least two potential issues:
1) your secrets are of type "opaque" meaning I do not think they were created properly. (they should be type docker). See:
for instructions on how to create proper docker secrets. Specifically I would recommend this syntax, given that you have a .docker/config.json format:
$ oc create secret generic <pull_secret_name> \
I do not believe the secret needs to be linked to anything for it to be used for to work.
2) your imagestream is marked w/ reference policy source, so they are not enabled for pullthrough. That's not causing the import errors, but it's going to cause you problems once you get through the import error
Miheer, please let us know the result of my suggestions above.
(In reply to Ben Parees from comment #13)
> Miheer, bump.
creating the secret with explicit type docker solved the registry authorization problem for the image stream.
Great, so can this be closed?