Bug 1668233

Summary:	oc image mirror fails with error 'tag: unexpected end of JSON input' when trying to mirror image from Red Hat registry.
Product:	OpenShift Container Platform	Reporter:	Mahesh Taru <mtaru>
Component:	ImageStreams	Assignee:	Ben Parees <bparees>
Status:	CLOSED ERRATA	QA Contact:	XiuJuan Wang <xiuwang>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, bparees, jokerman, mmccomas, wzheng
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: oc image mirror command could not handle metadata coming from the redhat registry Consequence: images could not be mirrored from that registry this the oc image mirror tool Fix: metadata handling was updated Result: images can now be mirrored from the redhat registry	Story Points:	---
Clone Of:
Clones:	1670551 1676404 (view as bug list)		Environment:
Last Closed:	2019-06-04 10:42:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1670551, 1676404

Description Mahesh Taru 2019-01-22 09:26:37 UTC

Description of problem:
When trying to mirror image from Red Hat registry 'registry.access.redhat.com' or 'registry.redhat.io' it fails with error 'tag: unexpected end of JSON input'.

Version-Release number of selected component (if applicable):
3.9

How reproducible:
Always

Steps to Reproduce:
On OCP 3.9 try to mirror image from Red Hat registry.

# oc image mirror registry.access.redhat.com/openshift3/ose-ansible:v3.9.43 docker-registry-default.svc/openshift/ose-ansible:v3.9.43 --loglevel=10

Actual results:
Fails with error:
*****************************************
I0122 14:52:40.150579   83155 plugins.go:56] Registering credential provider: .dockercfg
I0122 14:52:40.151199   83155 round_trippers.go:417] curl -k -v -XGET  https://registry.access.redhat.com/v2/
I0122 14:52:46.649935   83155 round_trippers.go:436] GET https://registry.access.redhat.com/v2/ 200 OK in 6498 milliseconds
I0122 14:52:46.650108   83155 round_trippers.go:442] Response Headers:
I0122 14:52:46.650150   83155 round_trippers.go:445]     Server: Apache
I0122 14:52:46.650174   83155 round_trippers.go:445]     Content-Length: 2
I0122 14:52:46.650197   83155 round_trippers.go:445]     Cache-Control: max-age=0, no-cache, no-store
I0122 14:52:46.650220   83155 round_trippers.go:445]     Connection: keep-alive
I0122 14:52:46.650255   83155 round_trippers.go:445]     Docker-Distribution-Api-Version: registry/2.0
I0122 14:52:46.650278   83155 round_trippers.go:445]     Content-Type: application/json
I0122 14:52:46.650302   83155 round_trippers.go:445]     Expires: Tue, 22 Jan 2019 09:22:46 GMT
I0122 14:52:46.650325   83155 round_trippers.go:445]     Pragma: no-cache
I0122 14:52:46.650349   83155 round_trippers.go:445]     Date: Tue, 22 Jan 2019 09:22:46 GMT
I0122 14:52:46.653502   83155 round_trippers.go:417] curl -k -v -XHEAD  -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" https://registry.access.redhat.com/v2/openshift3/ose-ansible/manifests/v3.9.43
I0122 14:52:48.098730   83155 round_trippers.go:436] HEAD https://registry.access.redhat.com/v2/openshift3/ose-ansible/manifests/v3.9.43 302 Moved Temporarily in 1445 milliseconds
I0122 14:52:48.098793   83155 round_trippers.go:442] Response Headers:
I0122 14:52:48.098808   83155 round_trippers.go:445]     Pragma: no-cache
I0122 14:52:48.098821   83155 round_trippers.go:445]     Date: Tue, 22 Jan 2019 09:22:48 GMT
I0122 14:52:48.098834   83155 round_trippers.go:445]     Connection: keep-alive
I0122 14:52:48.098847   83155 round_trippers.go:445]     Server: Apache
I0122 14:52:48.098860   83155 round_trippers.go:445]     Cache-Control: max-age=0, no-cache, no-store
I0122 14:52:48.098873   83155 round_trippers.go:445]     Content-Type: text/html; charset=utf-8
I0122 14:52:48.098886   83155 round_trippers.go:445]     Expires: Tue, 22 Jan 2019 09:22:48 GMT
I0122 14:52:48.098899   83155 round_trippers.go:445]     Content-Length: 525
I0122 14:52:48.098913   83155 round_trippers.go:445]     Location: https://registry.access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ose-ansible/manifests/list/v3.9.43
I0122 14:52:48.099080   83155 round_trippers.go:417] curl -k -v -XHEAD  -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Referer: https://registry.access.redhat.com/v2/openshift3/ose-ansible/manifests/v3.9.43" https://registry.access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ose-ansible/manifests/list/v3.9.43
I0122 14:52:48.289649   83155 round_trippers.go:436] HEAD https://registry.access.redhat.com/webassets/docker/content/dist/rhel/server/7/7Server/multiarch/ose/3/containers/registry/openshift3/ose-ansible/manifests/list/v3.9.43 200 OK in 190 milliseconds
I0122 14:52:48.289702   83155 round_trippers.go:442] Response Headers:
I0122 14:52:48.289714   83155 round_trippers.go:445]     Accept-Ranges: bytes
I0122 14:52:48.289724   83155 round_trippers.go:445]     Connection: keep-alive
I0122 14:52:48.289732   83155 round_trippers.go:445]     Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
I0122 14:52:48.289740   83155 round_trippers.go:445]     X-Docker-Size: -1
I0122 14:52:48.289748   83155 round_trippers.go:445]     Server: Apache
I0122 14:52:48.289756   83155 round_trippers.go:445]     Etag: "2f85eb4c77183389da10fe95a6871b57:1546651581"
I0122 14:52:48.289764   83155 round_trippers.go:445]     Last-Modified: Fri, 21 Sep 2018 17:33:24 GMT
I0122 14:52:48.289772   83155 round_trippers.go:445]     Content-Length: 464
I0122 14:52:48.289779   83155 round_trippers.go:445]     Date: Tue, 22 Jan 2019 09:22:48 GMT
F0122 14:52:48.290150   83155 helpers.go:119] error: unable to retrieve source image registry.access.redhat.com/openshift3/ose-ansible by tag: unexpected end of JSON input
*****************************************

Expected results:

Should be able to mirror image from Red Hat registry.

Additional info:
Works on OCP 3.7.

Comment 2 Ben Parees 2019-01-22 18:58:57 UTC

This command was experimental in v3.9.  The issue w/ mirroring from the redhat registry is being resolved in v4.0, the fix will not be backported since the command was experimental.

Comment 4 XiuJuan Wang 2019-01-24 09:52:11 UTC

This cmd works in ocp 4.0 
client: oc v4.0.0-0.143.0
server: 4.0.0-0.nightly-2019-01-23-024459

[xiuwang@dhcp-140-96 ~]$ oc image mirror  registry.access.redhat.com/openshift3/jenkins-2-rhel7:v3.11 docker.io/xiuwang/jenkins-2-rhel7:v3.11 
docker.io/
  xiuwang/jenkins-2-rhel7
    blobs:
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:d134b18b98b0d113b7b1194a60efceaa2c06eff41386d6c14b0e44bfe557eee8 1.194KiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:e30797a7050fb24792d750b433f63ee2200603447868989f69ea2d17ef06849d 4.568KiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:ca8b1d3d5d62df1071d4ac27a727690796de5e8554f9f57f98f21da458593b68 7.136MiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:836b6d169c26dc1bbb677887b271335e9c41713f18d83ae91a04d66bb5ff0188 32.02MiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82 72.21MiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:4aa86705ce54f5825010ccb2ae3465d909d4d66dff2834e39b155a52c9b850d2 100.4MiB
      registry.access.redhat.com/openshift3/jenkins-2-rhel7 sha256:07d56856146dee3dcf38db6769b9b08a09a1352dd40be60c63bec96f20ecea9e 327.5MiB
    manifests:
      sha256:9aee6b0423712b4bee3094b315dd42186e6776a4484ec17e20a83ab6f35e9859 -> v3.11
  stats: shared=0 unique=7 size=539.3MiB ratio=1.00

phase 0:
  docker.io xiuwang/jenkins-2-rhel7 blobs=7 mounts=0 manifests=1 shared=0

info: Planning completed in 7.45s
uploading: docker.io/library/xiuwang/jenkins-2-rhel7 sha256:07d56856146dee3dcf38db6769b9b08a09a1352dd40be60c63bec96f20ecea9e 327.5MiB
uploading: docker.io/library/xiuwang/jenkins-2-rhel7 sha256:4aa86705ce54f5825010ccb2ae3465d909d4d66dff2834e39b155a52c9b850d2 100.4MiB
uploading: docker.io/library/xiuwang/jenkins-2-rhel7 sha256:836b6d169c26dc1bbb677887b271335e9c41713f18d83ae91a04d66bb5ff0188 32.02MiB
uploading: docker.io/library/xiuwang/jenkins-2-rhel7 sha256:ca8b1d3d5d62df1071d4ac27a727690796de5e8554f9f57f98f21da458593b68 7.136MiB
uploading: docker.io/library/xiuwang/jenkins-2-rhel7 sha256:23113ae36f8e9d98b1423e44673979132dec59db2805e473e931d83548b0be82 72.21MiB
sha256:9aee6b0423712b4bee3094b315dd42186e6776a4484ec17e20a83ab6f35e9859 docker.io/library/xiuwang/jenkins-2-rhel7:v3.11
info: Mirroring completed in 3m26.62s (2.737MB/s)

Comment 7 Ben Parees 2019-02-12 14:40:36 UTC

*** Bug 1676404 has been marked as a duplicate of this bug. ***

Comment 10 errata-xmlrpc 2019-06-04 10:42:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758