Bug 1898669

Summary:	[sig-builds][Feature:Builds] Multi-stage image builds should succeed
Product:	OpenShift Container Platform	Reporter:	Hongkai Liu <hongkliu>
Component:	Build	Assignee:	Gabe Montero <gmontero>
Status:	CLOSED DUPLICATE	QA Contact:	XiuJuan Wang <xiuwang>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	aos-bugs
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:	[sig-builds][Feature:Builds] Multi-stage image builds should succeed
Last Closed:	2020-11-17 21:09:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hongkai Liu 2020-11-17 19:05:58 UTC

test:
[sig-builds][Feature:Builds] Multi-stage image builds  should succeed 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-builds%5C%5D%5C%5BFeature%3ABuilds%5C%5D+Multi-stage+image+builds++should+succeed


https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.5-e2e-vsphere/1328744712613400576

fail [github.com/openshift/origin/test/extended/builds/multistage.go:79]: Build did not succeed: &util.BuildResult{BuildPath:"builds/multi-stage", BuildName:"multi-stage", StartBuildStdErr:"", StartBuildStdOut:"", StartBuildErr:error(nil), BuildConfigName:"", Build:(*v1.Build)(0xc00215ea80), BuildAttempt:true, BuildSuccess:false, BuildFailure:true, BuildCancelled:false, BuildTimeout:false, LogDumper:(util.LogDumperFunc)(nil), Oc:(*util.CLI)(0xc0003dfe00)}
Expected
    <bool>: false
to be true

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-fips-4.5/1328657128855965696

fail [github.com/openshift/origin/test/extended/util/client.go:693]: Nov 17 11:51:55.042: Get https://api.ci-op-wyzpggwb-e0073.origin-ci-int-aws.dev.rhcloud.com:6443/apis/user.openshift.io/v1/users/~: dial tcp 52.13.136.230:6443: connect: connection refused

Comment 1 Adam Kaplan 2020-11-17 20:00:07 UTC

Seeing DockerHub-related rate limit throttling:

[sig-builds][Feature:Builds] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting [Suite:openshift/conformance/parallel]
2020-11-17T18:05:02.641436603Z error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://busybox" as "busybox": Error initializing source docker://busybox:latest: Error reading manifest latest in docker.io/library/busybox: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[sig-builds][Feature:Builds] Optimized image builds should succeed [Suite:openshift/conformance/parallel]
error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://centos:7" as "centos:7": Error initializing source docker://centos:7: Error reading manifest 7 in docker.io/library/centos: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [Suite:openshift/conformance/parallel] expand_less
container image registry lookup failed: docker.io/openshift/hello-openshift:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[sig-builds][Feature:Builds][valueFrom] process valueFrom in build strategy environment variables should successfully resolve valueFrom in docker build environment variables [Suite:openshift/conformance/parallel]


[sig-builds][Feature:Builds] custom build with buildah being created from new-build should complete build with custom builder image [Suite:openshift/conformance/parallel]
error creating build container: Error initializing source docker://centos:7: Error reading manifest 7 in docker.io/library/centos: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Comment 2 Gabe Montero 2020-11-17 21:09:53 UTC

Clayton wants to solve this with the work he is doing with https://bugzilla.redhat.com/show_bug.cgi?id=1816812

And in fact if you look at the associated PR https://github.com/openshift/origin/pull/24887 and its test history https://prow.ci.openshift.org/pr-history?org=openshift&repo=origin&pr=24887
e2e-*-builds has passes when k8s level flakes have not intervened.

For good measure, there is also the defect https://bugzilla.redhat.com/show_bug.cgi?id=1895107 which also covers this docker.io throttling.  In a sense it is serving as the backup
for 1816812.

I'm closing this one as a dupe as I am already monitoring / involved in these other 2 bz's.

*** This bug has been marked as a duplicate of bug 1816812 ***