Bug 2053222 - ImagePull fails with error "unable to pull manifest from example.com/busy.box:v5 invalid reference format"
Summary: ImagePull fails with error "unable to pull manifest from example.com/busy.bo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.9.z
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On: 2053218
Blocks: 2053223
TreeView+ depends on / blocked
 
Reported: 2022-02-10 17:25 UTC by OpenShift BugZilla Robot
Modified: 2022-03-07 08:28 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-23 20:02:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift image-registry pull 309 0 None Merged Bug 2053222: Fix pull-through for images that have dots in their namespace 2022-02-13 17:58:43 UTC
Github openshift library-go pull 1311 0 None Merged [release-4.9] Bug 2053222: Fix handling namespaces with dots in registry client 2022-02-11 16:08:09 UTC
Github openshift oc pull 1068 0 None Merged Bug 2053222: Fix mirroring images that have dots in their namespace 2022-02-14 10:04:28 UTC
Github openshift openshift-apiserver pull 282 0 None Merged Bug 2053222: Fix importing images that have dots in their namespace 2022-02-16 17:40:25 UTC
Red Hat Product Errata RHSA-2022:0561 0 None None None 2022-02-23 20:03:15 UTC

Comment 1 XiuJuan Wang 2022-02-12 07:30:57 UTC
Verified on pr premerged.
oc version
Client Version: v4.2.0-alpha.0-1202-g052fff7
Server Version: 4.9.0-0.ci.test-2022-02-12-053520-ci-ln-95hss3k-latest
Kubernetes Version: v1.22.1-1839+b93fd35dd03051-dirty

$oc image mirror quay.io/rh-test.me/busybox:latest $registry/test.myimage.a.b.c:latest --keep-manifest-list=true --filter-by-os='.*' --insecure

registry-wxj.apps.ci-ln-gxt32yk-72292.origin-ci-int-gce.dev.rhcloud.com/
  test.myimage.a.b.c.d
    blobs:
      quay.io/rh-test.me/busybox sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710 1.422KiB
      quay.io/rh-test.me/busybox sha256:009932687766e1520a47aa9de3bfe97ffdb1b6cad0b08d5078bad60329f13f19 754.7KiB
    manifests:
      sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578 -> latest
  stats: shared=0 unique=2 size=756.1KiB ratio=1.00

phase 0:
  registry-wxj.apps.ci-ln-gxt32yk-72292.origin-ci-int-gce.dev.rhcloud.com test.myimage.a.b.c blobs=2 mounts=0 manifests=1 shared=0

info: Planning completed in 4.92s
uploading: registry-wxj.apps.ci-ln-gxt32yk-72292.origin-ci-int-gce.dev.rhcloud.com/test.myimage.a.b.c.d sha256:009932687766e1520a47aa9de3bfe97ffdb1b6cad0b08d5078bad60329f13f19 754.7KiB
sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578 registry-wxj.apps.ci-ln-gxt32yk-72292.origin-ci-int-gce.dev.rhcloud.com/test.myimage.a.b.c:latest
info: Mirroring completed in 5.12s (151.1kB/s)

$ oc import-image myimage:latest --from=$registry/test.myimage.a.b.c:latest --reference-policy=local  --insecure --confirm
$ oc set image-lookup myimage
$ oc run mypod --restart=Never --image=myimage:latest --image-pull-policy=Always -- echo "Hello"
pod/mypod created

  Normal   Pulling         12s               kubelet            Pulling image "image-registry.openshift-image-registry.svc:5000/wxj/myimage@sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578"
  Normal   Pulled          11s               kubelet            Successfully pulled image "image-registry.openshift-image-registry.svc:5000/wxj/myimage@sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578" in 989.273735ms

 $oc -n openshift-image-registry get pods -l docker-registry=default -o name | while read -r pod; do oc -n openshift-image-registry logs "$pod"; done | grep "test.myimage.a.b.c"
time="2022-02-12T07:28:32.123586525Z" level=info msg="Trying to stat \"sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710\" from \"registry-wxj.apps.wxjfastfix49.qe.devcluster.openshift.com/test.myimage.a.b.c\" with a fall-back to insecure transport" go.version=go1.16.12 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=bafd07cb-b0e0-43d1-9da2-e3c12f9cc740 http.request.method=GET http.request.remoteaddr="10.129.2.1:55484" http.request.uri="/v2/wxj/myimage/blobs/sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710" http.request.useragent="cri-o/1.22.1-16.rhaos4.9.git12fa1c7.el8 go/go1.16.6 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:wxj:default" vars.digest="sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710" vars.name=wxj/myimage

Comment 6 Peter Ruan 2022-02-16 20:48:21 UTC
I'm still seeing following @xiuwang's steps
Server Version: 4.9.0-0.nightly-2022-02-16-161609
Kubernetes Version: v1.22.3+b93fd35

### setup $registry env varible by doing:
$oc patch configs.imageregistry.operator.openshift.io/cluster -p='{"spec":{"defaultRoute":true}}' --type=merge
$oc create serviceaccount registry
$oc adm policy add-cluster-role-to-user admin -z registry
$oc extract secret/pull-secret -n openshift-config --to=- | jq > ./pull-secret.json
$podman login --tls-verify=false --authfile=./pull-secret.json $(oc get route -n openshift-image-registry -o jsonpath='{.items[*].spec.host}') -u registry -p $(oc sa get-token registry)

### actual steps taken to verify the fix
1. export registry="default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com"
2. oc image mirror quay.io/rh-test.me/busybox:latest $registry/test.myimage.a.b.c:latest --keep-manifest-list=true --filter-by-os='.*' --insecure

pruan@cube ~/tmp $ oc image mirror quay.io/rh-test.me/busybox:latest $registry/test.myimage.a.b.c:latest --keep-manifest-list=true --filter-by-os='.*' --insecure -a ./pull-secret.json
default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com/
  test.myimage.a.b.c
    blobs:
      quay.io/rh-test.me/busybox sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710 1.422KiB
      quay.io/rh-test.me/busybox sha256:009932687766e1520a47aa9de3bfe97ffdb1b6cad0b08d5078bad60329f13f19 754.7KiB
    manifests:
      sha256:b69959407d21e8a062e0416bf13405bb2b71ed7a84dde4158ebafacfa06f5578 -> latest
  stats: shared=0 unique=2 size=756.1KiB ratio=1.00

phase 0:
  default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com test.myimage.a.b.c blobs=2 mounts=0 manifests=1 shared=0

info: Planning completed in 1.46s
error: unable to upload blob sha256:009932687766e1520a47aa9de3bfe97ffdb1b6cad0b08d5078bad60329f13f19 to default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com/test.myimage.a.b.c: Post "https://default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com/v2/test.myimage.a.b.c/blobs/uploads/": unauthorized: repository name "test.myimage.a.b.c" invalid: it must be of the format <project>/<name>
error: unable to push quay.io/rh-test.me/busybox: failed to upload blob sha256:ec3f0931a6e6b6855d76b2d7b0be30e81860baccd891b2e243280bf1cd8ad710: Post "https://default-route-openshift-image-registry.apps.pruan-0216.qe.devcluster.openshift.com/v2/test.myimage.a.b.c/blobs/uploads/": unauthorized: repository name "test.myimage.a.b.c" invalid: it must be of the format <project>/<name>
info: Mirroring completed in 470ms (0B/s)
error: one or more errors occurred while uploading images

Comment 7 Oleg Bulatov 2022-02-16 21:37:00 UTC
It must be of the format <project>/<name>, but you provided only <project>. The integrated registry doesn't support such naming scheme.

Comment 8 XiuJuan Wang 2022-02-17 01:12:31 UTC
Hi peter,
Thanks for checking.
I used a setup registry but not intergrated registry.

The steps to validate
$./oc version
Client Version: 4.9.0-0.nightly-2022-02-16-161609
Server Version: 4.9.0-0.nightly-2022-02-16-161609
Kubernetes Version: v1.22.3+b93fd35
$oc new-app quay.io/openshifttest/registry@sha256:01493571d994fd021da18c1f87aba1091482df3fc20825f443b4e60b3416c820
$oc create route edge registry --service=registry
export registry=`oc get route registry -o json | jq -r .spec.host`
$./oc image mirror  quay.io/rh-test.me/busybox:latest $registry/wxj-rh.test/test-rh.abcd1:latest --keep-manifest-list=true --filter-by-os='.*' --insecure
./oc import-image myimage:latest --from=$registry/wxj-rh.test/test-rh.abcd1:latest  --reference-policy=local  --insecure --confirm
$./oc set image-lookup myimage
$./oc run mypod --restart=Never --image=myimage:latest --image-pull-policy=Always -- echo "Hello"

Comment 10 errata-xmlrpc 2022-02-23 20:02:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.22 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0561


Note You need to log in before you can comment on or make changes to this bug.