periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-e2e-compact-remote-libvirt-ppc64le periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-s390x is failing frequently in CI, see: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-e2e-compact-remote-libvirt-ppc64le Error: [36mINFO[0m[2021-08-09T04:56:54Z] standard_init_linux.go:219: exec user process caused: exec format error [36mINFO[0m[2021-08-09T04:58:17Z] Imported release 4.9.0-0.nightly-2021-08-07-175228 created at 2021-08-07 17:54:17 +0000 UTC with 141 images to tag release:latest [36mINFO[0m[2021-08-09T04:58:17Z] Ran for 1m36s [31mERRO[0m[2021-08-09T04:58:17Z] Some steps failed: [31mERRO[0m[2021-08-09T04:58:17Z] * could not run steps: step [release:s390x-latest] failed: failed to get CLI image: unable to find the 'cli' image in the provided release image: the pod ci-op-53q5qixg/release-images-s390x-latest-cli failed after 9s (failed containers: release): ContainerFailed one or more containers exited
Can you point out the periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-e2e-compact-remote-libvirt-ppc64le job which failed with the similar issue? Can you confirm if the job failing for ppc64le is https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-ppc64le ?
Sorry, I mistakenly listed compact. It's both of these jobs: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-ppc64le https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-s390x
The *s390x* job that is successful (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiar[…]4.9-ocp-installer-remote-libvirt-s390x/1424421723737952256) is pulling *BOTH* the s390x and ppc64le images... INFO[2021-08-08T17:26:22Z] Resolved release latest to registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-08-07-175228 INFO[2021-08-08T17:26:22Z] Resolved release ppc64le-initial to registry.ci.openshift.org/ocp-ppc64le/release-ppc64le:4.9.0-0.nightly-ppc64le-2021-08-07-155716 INFO[2021-08-08T17:26:22Z] Resolved release ppc64le-latest to registry.ci.openshift.org/ocp-ppc64le/release-ppc64le:4.9.0-0.nightly-ppc64le-2021-08-07-172251 INFO[2021-08-08T17:26:22Z] Resolved release s390x-initial to registry.ci.openshift.org/ocp-s390x/release-s390x:4.9.0-0.nightly-s390x-2021-08-07-155712 INFO[2021-08-08T17:26:22Z] Resolved release s390x-latest to registry.ci.openshift.org/ocp-s390x/release-s390x:4.9.0-0.nightly-s390x-2021-08-07-172256 INFO[2021-08-08T17:26:22Z] Resolved release initial to registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-08-06-170119 The *s390x* jobs that are failing (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiar[…]4.9-ocp-installer-remote-libvirt-s390x/1424595440749252608) are only pulling the *ppc64le* images... INFO[2021-08-09T04:56:41Z] Resolved release initial to registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-08-06-170119 INFO[2021-08-09T04:56:41Z] Resolved release latest to registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-08-07-175228 INFO[2021-08-09T04:56:41Z] Resolved release ppc64le-initial to registry.ci.openshift.org/ocp-ppc64le/release-ppc64le:4.9.0-0.nightly-ppc64le-2021-08-07-155716 INFO[2021-08-09T04:56:41Z] Resolved release ppc64le-latest to registry.ci.openshift.org/ocp-ppc64le/release-ppc64le:4.9.0-0.nightly-ppc64le-2021-08-07-172251 I checked the release-controller logic and it doesnt appear to have crossed the streams (s390x <-> ppc64le) anywhere. I have also verified that the *ppc64le* failures are caused by the same issue, except they are pulling the *s390x* images.
Here are the correct links (sorry)... Successful *s390x* job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-s390x/1424421723737952256 Failing *s390x* job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.9-ocp-installer-remote-libvirt-s390x/1424595440749252608
*** Bug 1991629 has been marked as a duplicate of this bug. ***
https://coreos.slack.com/archives/C01CQA76KMX/p1628505292268200
Setting "Blocker-" after chat with Deep
Hi Deep, do you think this bug will reach "ON_QA" by the end of this sprint (August 14th)? If not, we might want to add the "reviewed-in-sprint" flag.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759