Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1762139

Summary:	Failed to push image with proxy
Product:	OpenShift Container Platform	Reporter:	zhou ying <yinzhou>
Component:	Networking	Assignee:	Casey Callendrello <cdc>
Networking sub component:	openshift-sdn	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	adam.kaplan, aos-bugs, obulatov
Version:	4.2.0
Target Milestone:	---
Target Release:	4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-13 13:07:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description zhou ying 2019-10-16 03:00:45 UTC

Description of problem:
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-proxy-4.2/83

Version-Release number of selected component (if applicable):
release-openshift-ocp-installer-e2e-aws-proxy-4.2 


[Feature:Builds][Conformance] custom build with buildah being created from new-build should complete build with custom builder image [Suite:openshift/conformance/parallel/minimal] 

Pushing image image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest ...
Getting image source signatures
Copying blob sha256:39fe1b81aeca17f7ae3a62c557fa1b919473660efec90ed5f97be4a2e986557a
Copying blob sha256:9908e46907377e84bd6646bdb18abebeb4163b85135739e1cd60aae154d4557c
Copying blob sha256:114e48820ba1e7da74b69b1d59981bc1228889c2441853d6970c0d97c75a1d87
Copying blob sha256:5805e24eb844940a9bf5c3017fd6873b1ecc461c49055756bd20eb18e781bf12
Copying blob sha256:437fba92fcd369592d5ee1e60f7799272048e85b5af1c1b84d250f0df0c2639f
Copying blob sha256:92382625a8c8283baa5db7077e15ed2aa14686fa7ce61131c63257052b7c2fd6
Successfully pushed image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest
Warning: Push failed, retrying in 5s ...
Getting image source signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest
Warning: Push failed, retrying in 5s ...
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
error: build error: Failed to push image: error copying layers and metadata from "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.imagestore=/var/lib/shared]image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest" to "docker://image-registry.openshift-image-registry.svc:5000/e2e-test-custom-build-tpbdv/custom-builder-image:latest": Error trying to reuse blob sha256:9908e46907377e84bd6646bdb18abebeb4163b85135739e1cd60aae154d4557c at destination: failed to read from destination repository e2e-test-custom-build-tpbdv/custom-builder-image: 500 (Internal Server Error)
<----end of log for "custom-builder-image-1-build"/"docker-build"

Comment 1 Oleg Bulatov 2019-10-16 15:08:33 UTC

from the log https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-proxy-4.2/83/artifacts/e2e-aws-proxy/must-gather/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-34ff29512304f77b0ab70ea6850e7f8295a4d19e497ab690ea5102a7044ea993/namespaces/openshift-image-registry/pods/image-registry-6dc655f5f-4mfgc/registry/registry/logs/current.log:

s3aws: RequestError: send request failed
caused by: Put https://ci-op-88c62tki-9c5bf-c5mvm-image-registry-us-east-1-ajytqdyith.s3.amazonaws.com/docker/registry/v2/repositories/e2e-test-custom-build-tpbdv/custom-builder-image/_uploads/360b413c-8e93-4ebf-948b-1d8dff689dad/startedat: proxyconnect tcp: dial tcp 35.196.128.173:3128: connect: no route to host

Comment 2 Adam Kaplan 2019-11-05 14:40:10 UTC

@Oleg is there anything to fix here? Seems that the registry lost its connection to the storage backend - IMO returning 500 from the registry is the correct behavior.

Comment 3 Oleg Bulatov 2019-11-05 15:23:39 UTC

500 from the registry is OK, but I'm not sure about "no route to host" in CI

Comment 4 Casey Callendrello 2019-11-05 18:21:27 UTC

Question: is 35.196.128.173:3128 the address of the proxy, or the s3 endpoint?

Comment 5 zhou ying 2019-11-08 01:57:40 UTC

It's the address of the proxy.

Comment 6 Casey Callendrello 2019-11-13 13:07:35 UTC


*** This bug has been marked as a duplicate of bug 1765280 ***