Bug 1932643

Summary:	Feature:ImageAppend test sometime fails because of `unknown blob`
Product:	OpenShift Container Platform	Reporter:	Oleg Bulatov <obulatov>
Component:	Image Registry	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED DUPLICATE	QA Contact:	Wenjing Zheng <wzheng>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	aos-bugs, rmarasch
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-03-08 15:10:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Oleg Bulatov 2021-02-24 20:05:41 UTC

Description of problem:

test: [sig-imageregistry][Feature:ImageAppend] Image append should create images by appending them [Suite:openshift/conformance/parallel]

The test sometimes fails due to errors likes

error: uploading the source layer sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b failed: Patch "https://image-registry.openshift-image-registry.svc:5000/v2/e2e-test-image-append-x2z6w/test/blobs/uploads/255c02bc-0686-4885-804c-cabdd3e5a24e?_state=OuRDGVptJzYwiBi7cJ7OLJtE6UUrcgGFxnJqbZXFMOF7Ik5hbWUiOiJlMmUtdGVzdC1pbWFnZS1hcHBlbmQteDJ6NncvdGVzdCIsIlVVSUQiOiIyNTVjMDJiYy0wNjg2LTQ4ODUtODA0Yy1jYWJkZDNlNWEyNGUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjEtMDItMTlUMTI6MDM6MTAuMDY5NTc0MDAxWiJ9": unknown blob

(This error can be found in the test output.)

In the registry logs I can see:

time="2021-02-19T12:03:10.440054187Z" level=error msg="client disconnected during blob PATCH" contentLength=-1 copied=0 error="stream error: stream ID 51; CANCEL" go.version=go1.15.5 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=4b1455da-ceec-4d7a-8ee8-c653a904636f http.request.method=PATCH http.request.remoteaddr="10.128.2.17:60804" http.request.uri="/v2/e2e-test-image-append-x2z6w/test/blobs/uploads/255c02bc-0686-4885-804c-cabdd3e5a24e?_state=OuRDGVptJzYwiBi7cJ7OLJtE6UUrcgGFxnJqbZXFMOF7Ik5hbWUiOiJlMmUtdGVzdC1pbWFnZS1hcHBlbmQteDJ6NncvdGVzdCIsIlVVSUQiOiIyNTVjMDJiYy0wNjg2LTQ4ODUtODA0Yy1jYWJkZDNlNWEyNGUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjEtMDItMTlUMTI6MDM6MTAuMDY5NTc0MDAxWiJ9" http.request.useragent=Go-http-client/2.0 openshift.auth.user="system:serviceaccount:e2e-test-image-append-x2z6w:builder" vars.name=e2e-test-image-append-x2z6w/test vars.uuid=255c02bc-0686-4885-804c-cabdd3e5a24e

Version-Release number of selected component (if applicable):

I found it in 4.7 and 4.8.

How reproducible:

Sometimes.

Additional info:

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.8/1362724429691359232

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.8/1364144090546966528

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-ovn-4.8/1364562084272017408

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-ovn-4.8/1364484774214766592

Comment 1 Oleg Bulatov 2021-02-24 20:37:46 UTC

oc wanted to mirror the blob sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b from image-registry.openshift-image-registry.svc:5000/openshift/tools to image-registry.openshift-image-registry.svc:5000/e2e-test-image-append-x2z6w/test.

openshift/tools:latest is an image imported from quay.io.

From the logs [1] I can see that the registry tried to use pull-through for the blob, but failed due to 429:

time="2021-02-19T12:03:10.435117604Z" level=error msg="Error statting blob sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b in remote repository \"quay.io/openshift-release-dev/ocp-v4.0-art-dev\": Head \"https://quay.io/v2/openshift-release-dev/ocp-v4.0-art-dev/blobs/sha256:cca21acb641a96561e0cf9a0c1c7b7ffbaaefc92185bd8a9440f6049c838e33b\": error parsing HTTP 429 response body: invalid character '<' looking for beginning of value: \"<html>\\r\\n<head><title>429 Too Many Requests</title></head>\\r\\n<body bgcolor=\\\"white\\\">\\r\\n<center><h1>429 Too Many Requests</h1></center>\\r\\n<hr><center>nginx/1.12.1</center>\\r\\n</body>\\r\\n</html>\\r\\n\"" go.version=go1.15.5 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=503b9144-5a74-4646-84a8-9ce4017d45e9 http.request.method=GET http.request.remoteaddr="10.128.2.17:60804"

So the registry said "blob unknown to registry" for this blob in openshift/tools.

[1]: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.8/1362724429691359232/artifacts/e2e-aws/pods/openshift-image-registry_image-registry-9c76c4dcf-g9d7m_registry.log

Comment 3 Oleg Bulatov 2021-03-08 15:10:27 UTC


*** This bug has been marked as a duplicate of bug 1923536 ***