OpenShift-on-OpenStack 4.9 Proxy tests are permafailing[1]. The test consistently failing is: "[sig-imageregistry][Serial][Suite:openshift/registry/serial] Image signature workflow can push a signed image to openshift registry and verify it [Suite:openshift/conformance/serial]". The same test is consistently succeeding on OCP v4.10. This possibly is a follow-up to Bug 2041358 [1]: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.9-e2e-openstack-proxy
Looking at the test output from a more recent run, I'm observing several FailedMount events for Secrets and ConfigMaps that should otherwise exist: ``` Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:11 +0000 UTC - event for signer-1: {build-controller } BuildStarted: Build e2e-test-registry-signing-shx8g/signer-1 is now running Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1: {build-controller } BuildFailed: Build e2e-test-registry-signing-shx8g/signer-1 failed Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-proxy-ca-bundles" : object "e2e-test-registry-signing-shx8g"/"signer-1-global-ca" not registered Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "builder-dockercfg-cg6xg-push" : object "e2e-test-registry-signing-shx8g"/"builder-dockercfg-cg6xg" not registered Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "kube-api-access-22sxt" : [object "e2e-test-registry-signing-shx8g"/"kube-root-ca.crt" not registered, object "e2e-test-registry-signing-shx8g"/"openshift-service-ca.crt" not registered] Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-system-configs" : object "e2e-test-registry-signing-shx8g"/"signer-1-sys-config" not registered Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "builder-dockercfg-cg6xg-pull" : object "e2e-test-registry-signing-shx8g"/"builder-dockercfg-cg6xg" not registered Mar 1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-ca-bundles" : object "e2e-test-registry-signing-shx8g"/"signer-1-ca" not registered ``` Unclear if this is a Storage or Node related issue.
Note that the fix for this BZ merely adds additional debug information to the given test on failure. The change needs to be backported to 4.9 so we can further investigate why the build pod is starting without the cluster's CA trust bundle.
@adam.kaplan Is this issue really on QA. As per my understanding we are first observing the behaviour with extra debug statement and don't have the fix yet, no?
We don't have a fix for the root issue - that is being discussed in https://bugzilla.redhat.com/show_bug.cgi?id=2064800. This BZ was mainly for tracking the additional debug data that was dumped.
As this is additional debug statement. Original issue is yet to address Verified
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069