Bug 2009788

Summary:	Image-registry gets unauthorized from mirrors
Product:	OpenShift Container Platform	Reporter:	Apoorva Jagtap <apjagtap>
Component:	Image Registry	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED DUPLICATE	QA Contact:	XiuJuan Wang <xiuwang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.8	CC:	aos-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-07 10:54:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Apoorva Jagtap 2021-10-01 15:05:38 UTC

Description of problem:

[*] In a cluster with image mirroring configured, when a pod is created with image as `image-registry.openshift-image-registry.svc:5000/openshift/cli:latest` (example), the pod fails due to ImagePullBackOff.

- While analyzing the image-registry logs, it appears that somewhere while fetching the digest location by either search or cache, the digest fetched by image-registry pod is actually the `image-id` (as per podman inspect <image>).


[*] We have confirmed the output of `oc adm release mirror..` command, and that reports a successful completion.
- Some additional scenarios tested (referring digest as 'sha256:xyz'):

Scenario 1:
- When we specify the image as 'quay.io/<path>/<name>@sha256:xyz' within a pod, the pod comes up running as expected.

Scenario 2:
- When we specify the image as 'mirrored-registry/<path>/<name>@sha256:xyz' within the pod, it again comes up running as expected with correct digest.

However, this mismatch of digest is observed only when we specify the path of OCP's internal registry (image-registry.openshift-image-registry.svc:5000)

Version-Release number of selected component (if applicable):
- OCP v4.7.8


Actual results:
- Image pull fails.

Expected results:
- Image pulls should be successful.


Additional info:
- The CU have mirror registries configured in OCP v4.7 as well, but ever since upgrading to v4.8, they are encountering this issue.

Comment 3 Oleg Bulatov 2021-10-07 10:54:40 UTC

Both digests are correct.

The client gets an error when it tried to pull the blob sha256:84c9...5f. This is not an image digest, it's a blob digest (images consist of blobs). It gets an error because the image registry fails to proxy it from the mirror registry:

level=error msg="response completed with error" err.code=unknown err.detail="Get \"https://<REDACTED>/blobs/sha256:84c9<REDACTED>5f\": unauthorized: Not Authorized." err.message="unknown error"

This is because the registry's implementation of ICSP in 4.8 does not support authorization. The problem should be fixed in 4.9.

*** This bug has been marked as a duplicate of bug 1972009 ***