Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1792002

Summary: mirror job not pushing images, container setup failures
Product: OpenShift Container Platform Reporter: Jeff Peeler <jpeeler>
Component: Test InfrastructureAssignee: Steve Kuznetsov <skuznets>
Status: CLOSED WORKSFORME QA Contact:
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.3.zCC: wking
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-17 19:39:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Peeler 2020-01-16 20:42:53 UTC 
The jobs history for release-openshift-ocp-installer-e2e-aws-mirrors-4.3 seems to have about a 50% success rate, but both failures seen today were during container setup:

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.3/389

error: unable to push manifest to registry.svc.ci.openshift.org/ci-op-h8bgxxzv/release-scratch:4.3.0-0.nightly-2020-01-16-123848-baremetal-machine-controllers: received unexpected HTTP status: 504 Gateway Time-out
info: Mirroring completed in 5m17.38s (18.67MB/s)
error: one or more errors occurred while uploading images
ignore: the release could not be reproduced from its inputs
error: unable to retrieve release image info: image does not exist
2020/01/16 18:38:20 Container setup in pod e2e-aws failed, exit code 1, reason Error
2020/01/16 18:38:41 Releasing lease for "aws-quota-slice"
2020/01/16 18:43:26 No custom metadata found and prow metadata already exists. Not updating the metadata.
2020/01/16 18:43:28 Ran for 37m39s
error: could not run steps: step e2e-aws failed: template pod "e2e-aws" failed: the pod ci-op-h8bgxxzv/e2e-aws failed after 7m30s (failed containers: setup): ContainerFailed one or more containers exited

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.3/387

error: unable to push manifest to registry.svc.ci.openshift.org/ci-op-fbk7h8fy/release:4.3.0-0.nightly-2020-01-15-025207-kube-etcd-signer-server: manifest blob unknown: blob unknown to registry
info: Mirroring completed in 58.79s (8.872kB/s)
error: one or more errors occurred while uploading images
2020/01/16 02:19:10 Container setup in pod e2e-aws failed, exit code 1, reason Error
2020/01/16 02:20:11 Releasing lease for "aws-quota-slice"
2020/01/16 02:28:54 No custom metadata found and prow metadata already exists. Not updating the metadata.
2020/01/16 02:28:55 Ran for 25m26s
error: could not run steps: step e2e-aws failed: template pod "e2e-aws" failed: the pod ci-op-fbk7h8fy/e2e-aws failed after 6m23s (failed containers: setup): ContainerFailed one or more containers exited

The latter error seems related to https://bugzilla.redhat.com/show_bug.cgi?id=1786372 (but was reported in 4.2).
Comment 1 Steve Kuznetsov 2020-01-16 22:17:11 UTC 
This is generic internal OpenShift Registry issues ... we are generally powerless to make much of a change here, hopefully it resolves itself. Sometimes this happens when load is too high on the nodes running the registry.
Comment 2 Steve Kuznetsov 2020-01-17 19:39:00 UTC 
I restarted the registry pods and judging by the lack of noise here it seems to have worked, closing.
Comment 3 W. Trevor King 2020-02-06 20:16:19 UTC 
Also mitigated via bug 1754042.
Comment 4 W. Trevor King 2020-02-07 19:30:45 UTC 
Looks like this is also being discussed in bug 1531676.