Bug 2049234 - ImagePull fails with error "unable to pull manifest from example.com/busy.box:v5 invalid reference format"
Summary: ImagePull fails with error "unable to pull manifest from example.com/busy.bo...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 4.8
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
Depends On:
Blocks: 2053218
TreeView+ depends on / blocked
Reported: 2022-02-01 19:18 UTC by Divyam Pateriya
Modified: 2022-08-10 10:46 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-08-10 10:46:22 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 307 0 None Merged Bug 2049234: Fix pull-through for images that have dots in their namespace 2022-02-11 16:40:44 UTC
Github openshift library-go pull 1309 0 None Merged Bug 2049234: Fix handling namespaces with dots in registry client 2022-02-11 16:40:44 UTC
Github openshift oc pull 1063 0 None Merged Bug 2049234: Fix mirroring images that have dots in their namespace 2022-02-11 16:40:45 UTC
Github openshift openshift-apiserver pull 279 0 None Merged Bug 2049234: Fix importing images that have dots in their namespace 2022-02-11 16:40:46 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:46:54 UTC

Description Divyam Pateriya 2022-02-01 19:18:07 UTC
Description of problem: Pods referring to the internal openshift registry through an imagestream tag do not work when the referred image in the external registry has a dot in the name of its repo.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1.  Create an imagestream as below:

oc apply -f - <<EOF
apiVersion: image.openshift.io/v1
kind: ImageStream
  name: bb
  namespace: test-pat5
    local: true
  - from:
      kind: DockerImage
      name: example.com/busy.box:v5
    importPolicy: {}
    name: "v5a"
      type: Local

2. Deploy a pod that uses the imagestream tag name

# oc run p --restart=Never --image=bb:v5a --image-pull-policy=Always -- echo "Hello"

3. Check if the pod has successfully pulled the image or not.

# oc get events 

13s       Warning   Failed             pod/p                 Failed to pull image "image-registry.openshift-image-registry.svc:5000/test-pat5/bb@sha256:dd97a3fe6d721c5cf03abac0f50e2848dc583f7c4e41bf39102ceb42edfd1808": rpc error: code = Unknown desc = Error reading manifest sha256:dd97a3fe6d721c5cf03abac0f50e2848dc583f7c4e41bf39102ceb42edfd1808 in image-registry.openshift-image-registry.svc:5000/test-pat5/bb: unknown: unable to pull manifest from example.com/busy.box@sha256:dd97a3fe6d721c5cf03abac0f50e2848dc583f7c4e41bf39102ceb42edfd1808: invalid reference format

Actual results:

The image pull fails with the error "invalid reference format"

Expected results:

The image should be pulled successfully

Additional info:


Comment 8 XiuJuan Wang 2022-02-11 03:55:03 UTC
Checked on 4.11.0-0.nightly-2022-02-11-014337 cluster with oc client build from https://github.com/dmage/oc/tree/namespace-with-dot
$oc version
Client Version: v4.2.0-alpha.0-1379-g3e7d761
Server Version: 4.11.0-0.nightly-2022-02-11-014337
Kubernetes Version: v1.23.3+f14faf2

still could reproduce this issue.
Could mirror image with dot in namespace, but imagestream import still failed.

$oc image mirror quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd:latest --keep-manifest-list=true --filter-by-os='.*' --insecure

sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd:latest
info: Mirroring completed in 13.44s (776.8kB/s)

$oc import-image wxj:v1 --from=registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd:latest  --insecure --confirm
error: tag latest failed: Internal error occurred: registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd1:latest: invalid reference format
imagestream.image.openshift.io/wxj imported with errors

Name:			wxj
Namespace:		default
Created:		Less than a second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2022-02-11T03:50:33Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/default/wxj
Image Lookup:		local=false
Unique Images:		0
Tags:			1

  tagged from registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd1:latest
    will use insecure HTTPS or HTTP connections

  ! error: Import failed (InternalError): Internal error occurred: registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.abcd:latest: invalid reference format
      Less than a second ago

Comment 9 Oleg Bulatov 2022-02-11 09:32:39 UTC
4.11.0-0.nightly-2022-02-11-014337 doesn't have the fix, 4.11.0-0.nightly-2022-02-11-053650 is the first build with the fix. Moving back to POST.

Comment 10 XiuJuan Wang 2022-02-11 11:16:20 UTC
Thanks Oleg, I didn't notice https://github.com/openshift/openshift-apiserver/pull/279 not included in 4.11.0-0.nightly-2022-02-11-014337.

Verified on 4.11.0-0.nightly-2022-02-11-053650

$oc version
Client Version: v4.2.0-alpha.0-1379-g3e7d761
Server Version: 4.11.0-0.nightly-2022-02-11-053650

$oc image mirror quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage:latest --keep-manifest-list=true --filter-by-os='.*' --insecure
W0211 19:02:41.261565    2782 helpers.go:151] Defaulting of registry auth file to "${HOME}/.docker/config.json" is deprecated. The default will be switched to podman config locations in the future version.

sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1f registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage:latest
info: Mirroring completed in 12.33s (846.6kB/s)

$oc tag registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage:latest bb1:v5a --reference-policy=local --insecure

$oc import-image mytest:latest --from=registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage:latest --insecure --confirm --reference-policy=local
imagestream.image.openshift.io/mytest imported

Name:			mytest
Namespace:		wxj
Created:		Less than a second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2022-02-11T11:13:20Z
Image Repository:	image-registry.openshift-image-registry.svc:5000/wxj/mytest
Image Lookup:		local=false
Unique Images:		1
Tags:			1

  tagged from registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage:latest
    will use insecure HTTPS or HTTP connections
    prefer registry pullthrough when referencing this tag

  * registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019
      Less than a second ago

Image Name:	mytest:latest
Docker Image:	registry-default.apps.wxjbz211.qe.devcluster.openshift.com/test.myimage@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019
Name:		sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019
Created:	Less than a second ago
Annotations:	image.openshift.io/dockerLayersOrder=ascending
Image Size:	766.1kB in 1 layers
Layers:		764.7kB	sha256:e5d9363303ddee1686b203170d78283404e46a742d4c62ac251aae5acbda8df8
Image Created:	13 months ago
Author:		<none>
Arch:		amd64
Command:	sh
Working Dir:	<none>
User:		<none>
Exposes Ports:	<none>
Docker Labels:	<none>
Environment:	PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$oc set image-lookup mytest

$oc run mypod --restart=Never --image=mytest:latest --image-pull-policy=Always -- echo "Hello"
pod/mypod created

$oc describe pods mypod
Pulling image "image-registry.openshift-image-registry.svc:5000/wxj/mytest@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019"
  Normal   Pulled          5s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/wxj/mytest@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019" in 673.969493ms

Comment 13 XiuJuan Wang 2022-02-12 06:31:55 UTC
Verified on 4.11.0-0.nightly-2022-02-11-215438 cluster

./oc version
Client Version: 4.11.0-0.nightly-2022-02-11-215438
Server Version: 4.11.0-0.nightly-2022-02-11-215438
Kubernetes Version: v1.23.3+f14faf2

Comment 15 errata-xmlrpc 2022-08-10 10:46:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.