Bug 1768507 - misleading image reference when pull-through fails
Summary: misleading image reference when pull-through fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.3.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-04 15:51 UTC by Oleg Bulatov
Modified: 2020-01-23 11:10 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-23 11:10:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 207 0 'None' closed Bug 1768507: keep digest when image is not referenced by tag 2021-02-12 09:31:01 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:10:51 UTC

Description Oleg Bulatov 2019-11-04 15:51:46 UTC
from 1718729#c2

The error message doesn't contain the image digest when pull-through fails.

> Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/jenkins@sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3": rpc error: code = Unknown desc = Error reading manifest sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3 in image-registry.openshift-image-registry.svc:5000/openshift/jenkins: unknown: unable to pull manifest from quay.io/openshift/origin-jenkins:latest: manifest unknown: manifest unknown

> In particular the internal registry reported "Unable to pull manifest from quay.io/openshift/origin-jenkins:latest" while doing the pullthrough, but the image we were trying to pull was quay.io/openshift/origin-jenkins@sha256:a85918b3631b200a14351c843b47cc19eba247ae56ad433585adf468b9994ae3.  Is there a reason the error reports "latest" not the actual sha we were trying to pull?  It leads to confusion since "latest" can indeed be pulled.

Comment 2 Wenjing Zheng 2019-11-11 10:48:23 UTC
I tried many times try to reproduce this bug, but all failed, could you give some hints on how to verify it? Thanks!

Comment 3 Oleg Bulatov 2019-11-11 17:01:21 UTC
I used these steps:

1. create a private repo on Docker Hub
2. create a secret for this repo
3. import an image from this repo with --reference-policy=local
4. delete the secret
5. try to pull this image from the integrated registry

Comment 4 Oleg Bulatov 2019-11-11 17:03:51 UTC
On step 3 I used the digest, but I'm not sure if it necessary:

oc tag --reference-policy=local <private_repo>/<image>@sha256:<digest> <imagestream>:<tag>

Comment 5 Wenjing Zheng 2019-11-12 08:12:00 UTC
Olge, I only can get below error when following your step:
Events:
  Type     Reason                  Age                From                                                 Message
  ----     ------                  ----               ----                                                 -------
  Normal   Scheduled               <unknown>          default-scheduler                                    Successfully assigned wzhengtest1/jenkins-1-m49rc to ip-10-0-147-36.ap-south-1.compute.internal
  Normal   SuccessfulAttachVolume  38s                attachdetach-controller                              AttachVolume.Attach succeeded for volume "pvc-1a11e5c1-c330-43db-8dbe-59593988eb6f"
  Normal   BackOff                 25s                kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d"
  Warning  Failed                  25s                kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Error: ImagePullBackOff
  Normal   Pulling                 12s (x2 over 27s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Pulling image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d"
  Warning  Failed                  11s (x2 over 26s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Failed to pull image "image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d": rpc error: code = Unknown desc = Error reading manifest sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d in image-registry.openshift-image-registry.svc:5000/wzhengtest1/jenkins: unknown: unable to pull manifest from quay.io/wzheng/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d: unauthorized: access to the requested resource is not authorized
  Warning  Failed                  11s (x2 over 26s)  kubelet, ip-10-0-147-36.ap-south-1.compute.internal  Error: ErrImagePull

Comment 6 Wenjing Zheng 2019-11-12 08:29:29 UTC
Some time I even can get a successful deployment after delete the secret : (

Comment 7 Oleg Bulatov 2019-11-12 10:25:47 UTC
> unknown: unable to pull manifest from quay.io/wzheng/jenkins@sha256:b1f624ef41d1ee3950f32631161895bd0f83447611c568f50e9283ff7d0bbd1d

You've verified it. On a previous version you would see jenkins:latest.

Comment 8 Wenjing Zheng 2019-11-12 10:51:59 UTC
OK, thanks for confirming, Oleg !!
Verified on 4.3.0-0.nightly-2019-11-11-182924

Comment 10 errata-xmlrpc 2020-01-23 11:10:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.