Bug 1768507

Summary: misleading image reference when pull-through fails
Product: OpenShift Container Platform Reporter: Oleg Bulatov <obulatov>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.3.0CC: aos-bugs
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-23 11:10:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Oleg Bulatov 2019-11-04 15:51:46 UTC
from 1718729#c2

The error message doesn't contain the image digest when pull-through fails.

> Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3": rpc error: code = Unknown desc = Error reading manifest sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3 in image-registry.openshift-image-registry.svc:5000/openshift/jenkins: unknown: unable to pull manifest from quay.io/openshift/origin-jenkins:latest: manifest unknown: manifest unknown

> In particular the internal registry reported "Unable to pull manifest from quay.io/openshift/origin-jenkins:latest" while doing the pullthrough, but the image we were trying to pull was quay.io/openshift/origin-jenkins@sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3.  Is there a reason the error reports "latest" not the actual sha we were trying to pull?  It leads to confusion since "latest" can indeed be pulled.

Comment 2 Wenjing Zheng 2019-11-11 10:48:23 UTC
I tried many times try to reproduce this bug, but all failed, could you give some hints on how to verify it? Thanks!

Comment 3 Oleg Bulatov 2019-11-11 17:01:21 UTC
I used these steps:

1. create a private repo on Docker Hub
2. create a secret for this repo
3. import an image from this repo with --reference-policy=local
4. delete the secret
5. try to pull this image from the integrated registry

Comment 4 Oleg Bulatov 2019-11-11 17:03:51 UTC
On step 3 I used the digest, but I'm not sure if it necessary:

oc tag --reference-policy=local <private_repo>/<image>@sha256:<digest> <imagestream>:<tag>

Comment 5 Wenjing Zheng 2019-11-12 08:12:00 UTC
Olge, I only can get below error when following your step:
Events:
  Type     Reason                  Age                From                                                 Message
  ----     ------                  ----               ----                                                 -------
  Normal   Scheduled               <unknown>          default-scheduler                                    Successfully assigned wzhengtest1/jenkins-1-m49rc to ip-10-0-147-36.ap-south-1.compute.internal
  Normal   SuccessfulAttachVolume  38s                attachdetach-controller                              AttachVolume.Attach succeeded for volume "pvc-1a11e5c1-c330-43db-8dbe-59593988eb6f"
  Normal   BackOff                 25s                kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d"
  Warning  Failed                  25s                kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Error: ImagePullBackOff
  Normal   Pulling                 12s (x2 over 27s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Pulling image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d"
  Warning  Failed                  11s (x2 over 26s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Failed to pull image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d": rpc error: code = Unknown desc = Error reading manifest sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d in image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins: unknown: unable to pull manifest from quay.io/wzheng/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d: unauthorized: access to the requested resource is not authorized
  Warning  Failed                  11s (x2 over 26s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Error: ErrImagePull

Comment 6 Wenjing Zheng 2019-11-12 08:29:29 UTC
Some time I even can get a successful deployment after delete the secret : (

Comment 7 Oleg Bulatov 2019-11-12 10:25:47 UTC
> unknown: unable to pull manifest from quay.io/wzheng/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d

You've verified it. On a previous version you would see jenkins:latest.

Comment 8 Wenjing Zheng 2019-11-12 10:51:59 UTC
OK, thanks for confirming, Oleg !!
Verified on 4.3.0-0.nightly-2019-11-11-182924

Comment 10 errata-xmlrpc 2020-01-23 11:10:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062