Bug 1692474
| Summary: | Image mirror fails with error 'tag invalid: manifest tag did not match URI' | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chris Callegari <ccallega> | ||||||||||
| Component: | oc | Assignee: | Clayton Coleman <ccoleman> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Wenjing Zheng <wzheng> | ||||||||||
| Severity: | low | Docs Contact: | |||||||||||
| Priority: | medium | ||||||||||||
| Version: | 4.1.0 | CC: | aos-bugs, ccoleman, jokerman, maszulik, mfojtik, mmccomas, vrutkovs, wking, wzheng, xxia | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | 4.4.0 | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2020-05-04 11:12:48 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Yeah this is me. Could just be a bad registry. I thought you may have had me. I was having trouble with the quay registry and took a couple days break but Maui is back! ;-) [root@ip-10-10-1-100 ~]# podman tag d8233ab899d4 registry.sysdeseng.com/abcd/openshift-4.0.0-0.7/busybox:latest [root@ip-10-10-1-100 ~]# podman push registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 Getting image source signatures Skipping fetch of repeat blob sha256:adab5d09ba79ecf30d3a5af58394b23a447eda7ffffe16c500ddc5ccb4c0222f Copying config sha256:d8233ab899d419c58cf3634c0df54ff5d8acc28f8173f09c21df4a07229e1205 0 B / 1.46 KB [------------------------------------------------------------] 0s Writing manifest to image destination Writing manifest to image destination Storing signatures [root@ip-10-10-1-100 ~]# podman push registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 Getting image source signatures Skipping fetch of repeat blob sha256:adab5d09ba79ecf30d3a5af58394b23a447eda7ffffe16c500ddc5ccb4c0222f Copying config sha256:d8233ab899d419c58cf3634c0df54ff5d8acc28f8173f09c21df4a07229e1205 0 B / 1.46 KB [------------------------------------------------------------] 0s Writing manifest to image destination Writing manifest to image destination Storing signatures Quay repository usage logs displays the pushes. CMD 1 - oc adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --mirror=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 --to-image=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 yields 'tag invalid: manifest tag did not match URI' errors CMD 2 - oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 yields also yields 'tag invalid: manifest tag did not match URI' errors Created attachment 1552519 [details]
oc_adm_release_cmd1.level10.log
oc --loglevel=10 adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --mirror=registry.sysdeseng.com/abcd/openshift4 --to-image=registry.sysdeseng.com/abcd/openshift4
Comment on attachment 1552519 [details]
oc_adm_release_cmd1.level10.log
oc --loglevel=10 adm release new --from-release=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --mirror=registry.sysdeseng.com/abcd/openshift4 --to-image=registry.sysdeseng.com/abcd/openshift4
Created attachment 1552520 [details]
oc_adm_release_cmd2.level10.log
oc --loglevel=10 adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.8 --to=registry.sysdeseng.com/abcd/openshift4
Registry reports the blobs exists:
I0405 13:51:12.282313 28456 glog.go:58] Server reports blob exists distribution.Descriptor{MediaType:"application/vnd.docker.container.image.rootfs.diff+x-gtar", Size:0, Digest:"sha256:d20ce882e6dcbd66fd274690fcdf461b5b648cba9b8050c1b9530102e0a91a4a", URLs:[]string(nil), Annotations:map[string]string(nil), Platform:(*v1.Platform)(nil)}
I0405 13:51:12.282403 28456 round_trippers.go:419] curl -k -v -XHEAD -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFjNGJlMmMyYjhiNzJiYWZlNWMzOTMyZjdkM2QyZGYzMTViMjczYWIwNzI3YjYzYjVlNWE5ZDUwOGE0MmQyZDkifQ.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImFiY2Qvb3BlbnNoaWZ0NCIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7ImVudGl0eV9raW5kIjoidXNlciIsImtpbmQiOiJ1c2VyIiwidmVyc2lvbiI6MiwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIiwidXNlciI6ImFiY2QiLCJlbnRpdHlfcmVmZXJlbmNlIjoiYWVmYzMxOGUtOTM1YS00MGU1LTg4NzYtYjE2MjhiYjJlNWExIiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJhYmNkL29wZW5zaGlmdDQiOiIkZGlzYWJsZWQifX0sImF1ZCI6InJlZ2lzdHJ5LnN5c2Rlc2VuZy5jb20iLCJleHAiOjE1NTQ0NzU4NjgsImlzcyI6InF1YXkiLCJpYXQiOjE1NTQ0NzIyNjgsIm5iZiI6MTU1NDQ3MjI2OCwic3ViIjoiYWJjZCJ9.UHCscm4K_VjDbB-mXccUP3ya_1v5t5GO8xA-VrZQ8yoLfmK6G5xZ4m80GtHpscvWs2kQNX7Wq1ekbQXk-UdpykHznE3uEpToxdWpJJb3JPgaChjTwv_cjwi8E4ZR6Wo2L1ly5TS-LPHImb8f_O0XZrBp5MRVTW32UJ37Is9QXo-sa-99qdc-xryV5ozNziPJNT8vsWBF-tgBC4p2P3tR5c6SCxS3YppJq6BWTlL--JXt3b46QoI6uJO8DTy1yf6wo1k2QR-cVEiROcRvfv5lKswhZOUM2_hB8PpZpPbb76n3j-POTD7riAvoG6k1D_MDi1c8bGStvUj8krw3KkcZkw" 'https://registry.sysdeseng.com/v2/abcd/openshift4/blobs/sha256:d4d95aa12ec9ebfcc050ee4e4bbce9add60c9578fc99859003bdd8cf98f54a11'
I0405 13:51:12.285151 28456 round_trippers.go:438] HEAD https://registry.sysdeseng.com/v2/abcd/openshift4/blobs/sha256:ea95f4abbba4bbf1c462125ab02d8e8515240c27ee862f93b9b45bc9b57a4e96 200 OK in 18 milliseconds
I0405 13:51:12.285168 28456 round_trippers.go:444] Response Headers:
I0405 13:51:12.285173 28456 round_trippers.go:447] Docker-Content-Digest: sha256:ea95f4abbba4bbf1c462125ab02d8e8515240c27ee862f93b9b45bc9b57a4e96
I0405 13:51:12.285179 28456 round_trippers.go:447] Cache-Control: max-age=31436000
I0405 13:51:12.285183 28456 round_trippers.go:447] X-Frame-Options: DENY
I0405 13:51:12.285188 28456 round_trippers.go:447] Strict-Transport-Security: max-age=63072000; preload
I0405 13:51:12.285192 28456 round_trippers.go:447] Server: nginx/1.13.12
I0405 13:51:12.285197 28456 round_trippers.go:447] Date: Fri, 05 Apr 2019 13:51:12 GMT
I0405 13:51:12.285202 28456 round_trippers.go:447] Content-Type: application/octet-stream
I0405 13:51:12.285206 28456 round_trippers.go:447] Content-Length: 32345528
but throws errors when a new tag is created:
I0405 13:51:12.716244 28456 glog.go:58] Put manifest abcd/openshift4:cluster-svcat-apiserver-operator
I0405 13:51:12.716320 28456 round_trippers.go:419] curl -k -v -XPUT -H "Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImFjNGJlMmMyYjhiNzJiYWZlNWMzOTMyZjdkM2QyZGYzMTViMjczYWIwNzI3YjYzYjVlNWE5ZDUwOGE0MmQyZDkifQ.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImFiY2Qvb3BlbnNoaWZ0NCIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7ImVudGl0eV9raW5kIjoidXNlciIsImtpbmQiOiJ1c2VyIiwidmVyc2lvbiI6MiwiY29tLmFwb3N0aWxsZS5yb290IjoiJGRpc2FibGVkIiwidXNlciI6ImFiY2QiLCJlbnRpdHlfcmVmZXJlbmNlIjoiYWVmYzMxOGUtOTM1YS00MGU1LTg4NzYtYjE2MjhiYjJlNWExIiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJhYmNkL29wZW5zaGlmdDQiOiIkZGlzYWJsZWQifX0sImF1ZCI6InJlZ2lzdHJ5LnN5c2Rlc2VuZy5jb20iLCJleHAiOjE1NTQ0NzU4NjgsImlzcyI6InF1YXkiLCJpYXQiOjE1NTQ0NzIyNjgsIm5iZiI6MTU1NDQ3MjI2OCwic3ViIjoiYWJjZCJ9.UHCscm4K_VjDbB-mXccUP3ya_1v5t5GO8xA-VrZQ8yoLfmK6G5xZ4m80GtHpscvWs2kQNX7Wq1ekbQXk-UdpykHznE3uEpToxdWpJJb3JPgaChjTwv_cjwi8E4ZR6Wo2L1ly5TS-LPHImb8f_O0XZrBp5MRVTW32UJ37Is9QXo-sa-99qdc-xryV5ozNziPJNT8vsWBF-tgBC4p2P3tR5c6SCxS3YppJq6BWTlL--JXt3b46QoI6uJO8DTy1yf6wo1k2QR-cVEiROcRvfv5lKswhZOUM2_hB8PpZpPbb76n3j-POTD7riAvoG6k1D_MDi1c8bGStvUj8krw3KkcZkw" 'https://registry.sysdeseng.com/v2/abcd/openshift4/manifests/cluster-svcat-apiserver-operator'
I0405 13:51:12.734202 28456 round_trippers.go:438] PUT https://registry.sysdeseng.com/v2/abcd/openshift4/manifests/machine-config-controller 400 Bad Request in 70 milliseconds
I0405 13:51:12.734231 28456 round_trippers.go:444] Response Headers:
I0405 13:51:12.734237 28456 round_trippers.go:447] Server: nginx/1.13.12
I0405 13:51:12.734242 28456 round_trippers.go:447] Date: Fri, 05 Apr 2019 13:51:12 GMT
I0405 13:51:12.734247 28456 round_trippers.go:447] Content-Type: application/json
I0405 13:51:12.734252 28456 round_trippers.go:447] Content-Length: 91
I just ran this command per our conversation in slack... oc adm release mirror \ --from=registry.svc.ci.openshift.org/openshift/origin-release:v4.0 \ --to=registry.sysdeseng.com/abcd/openshift4 The result is much better. This time I get lots of normal stuff then... error: unable to push manifest to registry.sysdeseng.com/abcd/openshift4:machine-os-content: tag invalid: manifest tag did not match URI info: Mirroring completed in 1m26.8s (30.49MB/s) error: one or more errors occurred while uploading images If I use the quay.io as the source then it's back to 81 tag invalid errors. This bugzilla appears to be resolved. Log is attached. Created attachment 1574509 [details]
oc_adm_release_new-success.log
Moving to qa based on one of previous comments which states it's fixed in recent versions. Verified with below command and image is mirrored successfully $ oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=upshift.mirror-registry.qe.devcluster.openshift.com:5000/wzheng/openshift-4.0.0-0.7 $ oc version Client Version: 4.4.0-202002282323-bc08a48 Kubernetes Version: v1.11.0+d4cacc0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
Created attachment 1547775 [details] oc command output Description of problem: Image mirror fails with error 'tag invalid: manifest tag did not match URI' Version-Release number of selected component (if applicable): $ oc version Client Version: version.Info{Major:"4", Minor:"0+", GitVersion:"v4.0.0-alpha.0+4bc0236-1762", GitCommit:"4bc0236", GitTreeState:"", BuildDate:"2019-03-23T10:32:09Z", GoVersion:"", Compiler:"", Platform:""} How reproducible: always Steps to Reproduce: 1. Deploy quay registry 2. Execute... oc adm release mirror --from=quay.io/openshift-release-dev/ocp-release:4.0.0-0.7 --to=registry.sysdeseng.com/abcd/openshift-4.0.0-0.7 3. Each image will get a similar error as this. Full log is attached. error: unable to push manifest to registry.sysdeseng.com/abcd/openshift-4.0.0-0.7:service-serving-cert-signer: tag invalid: manifest tag did not match URI Actual results: Image fails to push to private registry Expected results: Image is pushed to private registry correctly Additional info: