Bug 1692474 - Image mirror fails with error 'tag invalid: manifest tag did not match URI'
Summary: Image mirror fails with error 'tag invalid: manifest tag did not match URI'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.4.0
Assignee: Clayton Coleman
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-25 16:49 UTC by Chris Callegari
Modified: 2020-05-04 11:13 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:12:48 UTC
Target Upstream Version:


Attachments (Terms of Use)
oc command output (29.09 KB, text/plain)
2019-03-25 16:49 UTC, Chris Callegari
no flags Details
oc_adm_release_cmd1.level10.log (1.79 MB, text/plain)
2019-04-05 14:00 UTC, Chris Callegari
no flags Details
oc_adm_release_cmd2.level10.log (826.73 KB, text/plain)
2019-04-05 14:03 UTC, Chris Callegari
no flags Details
oc_adm_release_new-success.log (46.33 KB, text/plain)
2019-05-28 20:07 UTC, Chris Callegari
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:13:13 UTC

Description Chris Callegari 2019-03-25 16:49:48 UTC
Created attachment 1547775 [details]
oc command output

Description of problem:
Image mirror fails with error 'tag invalid: manifest tag did not match URI'

Version-Release number of selected component (if applicable):
$ oc version
Client Version: version.Info{Major:"4", Minor:"0+", GitVersion:"v4.0.0-alpha.0+4bc0236-1762", GitCommit:"4bc0236", GitTreeState:"", BuildDate:"2019-03-23T10:32:09Z", GoVersion:"", Compiler:"", Platform:""}


How reproducible:
always

Steps to Reproduce:
1. Deploy quay registry
2. Execute...
oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7
3. Each image will get a similar error as this.  Full log is attached.
error: unable to push manifest to registry.sysdeseng.com/abcd/openshift-4.0.0-0.7:service-serving-cert-signer: tag invalid: manifest tag did not match URI

Actual results:
Image fails to push to private registry

Expected results:
Image is pushed to private registry correctly 

Additional info:

Comment 2 Clayton Coleman 2019-03-26 15:38:43 UTC
Yeah this is me.  Could just be a bad registry.

Comment 3 Chris Callegari 2019-03-28 15:38:25 UTC
I thought you may have had me.  I was having trouble with the quay registry and took a couple days break but Maui is back! ;-)


[root@ip-10-10-1-100 ~]# podman tag d8233ab899d4 registry.sysdeseng.com/abcd/openshift-4.0.0-0.7/busybox:latest

[root@ip-10-10-1-100 ~]# podman push registry.sysdeseng.com/abcd/openshift-4.0.0-0.7
Getting image source signatures
Skipping fetch of repeat blob sha256:adab5d09ba79ecf30d3a5af58394b23a447eda7ffffe16c500ddc5ccb4c0222f
Copying config sha256:d8233ab899d419c58cf3634c0df54ff5d8acc28f8173f09c21df4a07229e1205
 0 B / 1.46 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
[root@ip-10-10-1-100 ~]# podman push registry.sysdeseng.com/abcd/openshift-4.0.0-0.7
Getting image source signatures
Skipping fetch of repeat blob sha256:adab5d09ba79ecf30d3a5af58394b23a447eda7ffffe16c500ddc5ccb4c0222f
Copying config sha256:d8233ab899d419c58cf3634c0df54ff5d8acc28f8173f09c21df4a07229e1205
 0 B / 1.46 KB [------------------------------------------------------------] 0s
Writing manifest to image destination
Writing manifest to image destination
Storing signatures

Quay repository usage logs displays the pushes.


CMD 1 - oc adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --mirror=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 --to-image=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7
yields 'tag invalid: manifest tag did not match URI' errors

CMD 2 - oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7
yields also yields 'tag invalid: manifest tag did not match URI' errors

Comment 4 Chris Callegari 2019-04-05 14:00:45 UTC
Created attachment 1552519 [details]
oc_adm_release_cmd1.level10.log

oc --loglevel=10 adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --mirror=registry.sysdeseng.com/abcd/openshift4 --to-image=registry.sysdeseng.com/abcd/openshift4

Comment 5 Chris Callegari 2019-04-05 14:01:40 UTC
Comment on attachment 1552519 [details]
oc_adm_release_cmd1.level10.log

oc --loglevel=10 adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --mirror=registry.sysdeseng.com/abcd/openshift4 --to-image=registry.sysdeseng.com/abcd/openshift4

Comment 6 Chris Callegari 2019-04-05 14:03:03 UTC
Created attachment 1552520 [details]
oc_adm_release_cmd2.level10.log

oc --loglevel=10 adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --to=registry.sysdeseng.com/abcd/openshift4

Comment 7 Vadim Rutkovsky 2019-04-10 18:48:20 UTC
Registry reports the blobs exists:

I0405 13:51:12.282313   28456 glog.go:58] Server reports blob exists distribution.Descriptor{MediaType:"application/vnd.docker.container.image.rootfs.diff+x-gtar", Size:0, Digest:"sha256:d20ce882e6dcbd66fd274690fcdf461b5b648cba9b8050c1b9530102e0a91a4a", URLs:[]string(nil), Annotations:map[string]string(nil), Platform:(*v1.Platform)(nil)}
I0405 13:51:12.282403   28456 round_trippers.go:419] curl -k -v -XHEAD  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFjNGJlMmMyYjhiNzJiYWZlNWMzOTMyZjdkM2QyZGYzMTViMjczYWIwNzI3YjYzYjVlNWE5ZDUwOGE0MmQyZDkifQ.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImFiY2Qvb3BlbnNoaWZ0NCIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7ImVudGl0eV9raW5kIjoidXNlciIsImtpbmQiOiJ1c2VyIiwidmVyc2lvbiI6MiwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIiwidXNlciI6ImFiY2QiLCJlbnRpdHlfcmVmZXJlbmNlIjoiYWVmYzMxOGUtOTM1YS00MGU1LTg4NzYtYjE2MjhiYjJlNWExIiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJhYmNkL29wZW5zaGlmdDQiOiIkZGlzYWJsZWQifX0sImF1ZCI6InJlZ2lzdHJ5LnN5c2Rlc2VuZy5jb20iLCJleHAiOjE1NTQ0NzU4NjgsImlzcyI6InF1YXkiLCJpYXQiOjE1NTQ0NzIyNjgsIm5iZiI6MTU1NDQ3MjI2OCwic3ViIjoiYWJjZCJ9.UHCscm4K_VjDbB-mXccUP3ya_1v5t5GO8xA-VrZQ8yoLfmK6G5xZ4m80GtHpscvWs2kQNX7Wq1ekbQXk-UdpykHznE3uEpToxdWpJJb3JPgaChjTwv_cjwi8E4ZR6Wo2L1ly5TS-LPHImb8f_O0XZrBp5MRVTW32UJ37Is9QXo-sa-99qdc-xryV5ozNziPJNT8vsWBF-tgBC4p2P3tR5c6SCxS3YppJq6BWTlL--JXt3b46QoI6uJO8DTy1yf6wo1k2QR-cVEiROcRvfv5lKswhZOUM2_hB8PpZpPbb76n3j-POTD7riAvoG6k1D_MDi1c8bGStvUj8krw3KkcZkw" 'https://registry.sysdeseng.com/v2/abcd/openshift4/blobs/sha256:d4d95aa12ec9ebfcc050ee4e4bbce9add60c9578fc99859003bdd8cf98f54a11'
I0405 13:51:12.285151   28456 round_trippers.go:438] HEAD https://registry.sysdeseng.com/v2/abcd/openshift4/blobs/sha256:ea95f4abbba4bbf1c462125ab02d8e8515240c27ee862f93b9b45bc9b57a4e96 200 OK in 18 milliseconds
I0405 13:51:12.285168   28456 round_trippers.go:444] Response Headers:
I0405 13:51:12.285173   28456 round_trippers.go:447]     Docker-Content-Digest: sha256:ea95f4abbba4bbf1c462125ab02d8e8515240c27ee862f93b9b45bc9b57a4e96
I0405 13:51:12.285179   28456 round_trippers.go:447]     Cache-Control: max-age=31436000
I0405 13:51:12.285183   28456 round_trippers.go:447]     X-Frame-Options: DENY
I0405 13:51:12.285188   28456 round_trippers.go:447]     Strict-Transport-Security: max-age=63072000; preload
I0405 13:51:12.285192   28456 round_trippers.go:447]     Server: nginx/1.13.12
I0405 13:51:12.285197   28456 round_trippers.go:447]     Date: Fri, 05 Apr 2019 13:51:12 GMT
I0405 13:51:12.285202   28456 round_trippers.go:447]     Content-Type: application/octet-stream
I0405 13:51:12.285206   28456 round_trippers.go:447]     Content-Length: 32345528

but throws errors when a new tag is created:

I0405 13:51:12.716244   28456 glog.go:58] Put manifest abcd/openshift4:cluster-svcat-apiserver-operator
I0405 13:51:12.716320   28456 round_trippers.go:419] curl -k -v -XPUT  -H "Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFjNGJlMmMyYjhiNzJiYWZlNWMzOTMyZjdkM2QyZGYzMTViMjczYWIwNzI3YjYzYjVlNWE5ZDUwOGE0MmQyZDkifQ.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImFiY2Qvb3BlbnNoaWZ0NCIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7ImVudGl0eV9raW5kIjoidXNlciIsImtpbmQiOiJ1c2VyIiwidmVyc2lvbiI6MiwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIiwidXNlciI6ImFiY2QiLCJlbnRpdHlfcmVmZXJlbmNlIjoiYWVmYzMxOGUtOTM1YS00MGU1LTg4NzYtYjE2MjhiYjJlNWExIiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJhYmNkL29wZW5zaGlmdDQiOiIkZGlzYWJsZWQifX0sImF1ZCI6InJlZ2lzdHJ5LnN5c2Rlc2VuZy5jb20iLCJleHAiOjE1NTQ0NzU4NjgsImlzcyI6InF1YXkiLCJpYXQiOjE1NTQ0NzIyNjgsIm5iZiI6MTU1NDQ3MjI2OCwic3ViIjoiYWJjZCJ9.UHCscm4K_VjDbB-mXccUP3ya_1v5t5GO8xA-VrZQ8yoLfmK6G5xZ4m80GtHpscvWs2kQNX7Wq1ekbQXk-UdpykHznE3uEpToxdWpJJb3JPgaChjTwv_cjwi8E4ZR6Wo2L1ly5TS-LPHImb8f_O0XZrBp5MRVTW32UJ37Is9QXo-sa-99qdc-xryV5ozNziPJNT8vsWBF-tgBC4p2P3tR5c6SCxS3YppJq6BWTlL--JXt3b46QoI6uJO8DTy1yf6wo1k2QR-cVEiROcRvfv5lKswhZOUM2_hB8PpZpPbb76n3j-POTD7riAvoG6k1D_MDi1c8bGStvUj8krw3KkcZkw" 'https://registry.sysdeseng.com/v2/abcd/openshift4/manifests/cluster-svcat-apiserver-operator'
I0405 13:51:12.734202   28456 round_trippers.go:438] PUT https://registry.sysdeseng.com/v2/abcd/openshift4/manifests/machine-config-controller 400 Bad Request in 70 milliseconds
I0405 13:51:12.734231   28456 round_trippers.go:444] Response Headers:
I0405 13:51:12.734237   28456 round_trippers.go:447]     Server: nginx/1.13.12
I0405 13:51:12.734242   28456 round_trippers.go:447]     Date: Fri, 05 Apr 2019 13:51:12 GMT
I0405 13:51:12.734247   28456 round_trippers.go:447]     Content-Type: application/json
I0405 13:51:12.734252   28456 round_trippers.go:447]     Content-Length: 91

Comment 8 Chris Callegari 2019-04-10 19:11:39 UTC
I just ran this command per our conversation in slack...
oc adm release mirror \
  --from=registry.svc.ci.openshift.org/openshift/origin-release:v4.0 \
  --to=registry.sysdeseng.com/abcd/openshift4

The result is much better.  This time I get lots of normal stuff then...
error: unable to push manifest to registry.sysdeseng.com/abcd/openshift4:machine-os-content: tag invalid: manifest tag did not match URI
info: Mirroring completed in 1m26.8s (30.49MB/s)
error: one or more errors occurred while uploading images



If I use the quay.io as the source then it's back to 81 tag invalid errors.

Comment 9 Chris Callegari 2019-05-28 20:06:30 UTC
This bugzilla appears to be resolved.  Log is attached.

Comment 10 Chris Callegari 2019-05-28 20:07:24 UTC
Created attachment 1574509 [details]
oc_adm_release_new-success.log

Comment 12 Maciej Szulik 2020-02-26 19:30:43 UTC
Moving to qa based on one of previous comments which states it's fixed in recent versions.

Comment 15 Wenjing Zheng 2020-03-03 05:47:13 UTC
Verified with below command and image is mirrored successfully
$ oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/wzheng/openshift-4.0.0-0.7
$ oc version
Client Version: 4.4.0-202002282323-bc08a48
Kubernetes Version: v1.11.0+d4cacc0

Comment 17 errata-xmlrpc 2020-05-04 11:12:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.