Bug 2057990

Summary: Add extra debug information to image signature workflow test
Product: OpenShift Container Platform Reporter: Pierre Prinetti <pprinett>
Component: BuildAssignee: Adam Kaplan <adam.kaplan>
Status: CLOSED ERRATA QA Contact: Jitendar Singh <jitsingh>
Severity: medium Docs Contact:
Priority: high    
Version: 4.9CC: adam.kaplan, obulatov, pbhattac, pkumari, spandura
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2064800 (view as bug list) Environment:
Last Closed: 2022-08-10 10:51:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2061554    

Description Pierre Prinetti 2022-02-24 09:27:15 UTC
OpenShift-on-OpenStack 4.9 Proxy tests are permafailing[1]. The  test consistently failing is: "[sig-imageregistry][Serial][Suite:openshift/registry/serial] Image signature workflow can push a signed image to openshift registry and verify it [Suite:openshift/conformance/serial]".

The same test is consistently succeeding on OCP v4.10.

This possibly is a follow-up to Bug 2041358

[1]: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.9-e2e-openstack-proxy

Comment 2 Adam Kaplan 2022-03-04 20:14:35 UTC
Looking at the test output from a more recent run, I'm observing several FailedMount events for Secrets and ConfigMaps that should otherwise exist:

```
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:11 +0000 UTC - event for signer-1: {build-controller } BuildStarted: Build e2e-test-registry-signing-shx8g/signer-1 is now running
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1: {build-controller } BuildFailed: Build e2e-test-registry-signing-shx8g/signer-1 failed
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-proxy-ca-bundles" : object "e2e-test-registry-signing-shx8g"/"signer-1-global-ca" not registered
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "builder-dockercfg-cg6xg-push" : object "e2e-test-registry-signing-shx8g"/"builder-dockercfg-cg6xg" not registered
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "kube-api-access-22sxt" : [object "e2e-test-registry-signing-shx8g"/"kube-root-ca.crt" not registered, object "e2e-test-registry-signing-shx8g"/"openshift-service-ca.crt" not registered]
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-system-configs" : object "e2e-test-registry-signing-shx8g"/"signer-1-sys-config" not registered
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "builder-dockercfg-cg6xg-pull" : object "e2e-test-registry-signing-shx8g"/"builder-dockercfg-cg6xg" not registered
Mar  1 20:41:09.639: INFO: At 2022-03-01 20:36:25 +0000 UTC - event for signer-1-build: {kubelet 4c3pnl1n-d61d5-9p5l4-worker-0-dnwjs} FailedMount: MountVolume.SetUp failed for volume "build-ca-bundles" : object "e2e-test-registry-signing-shx8g"/"signer-1-ca" not registered
```

Unclear if this is a Storage or Node related issue.

Comment 4 Adam Kaplan 2022-03-07 20:02:45 UTC
Note that the fix for this BZ merely adds additional debug information to the given test on failure. The change needs to be backported to 4.9 so we can further investigate why the build pod is starting without the cluster's CA trust bundle.

Comment 6 Priti Kumari 2022-04-08 07:26:41 UTC
@adam.kaplan Is this issue really on QA. As per my understanding we are first observing the behaviour with extra debug statement and don't have the fix yet, no?

Comment 7 Adam Kaplan 2022-04-08 19:36:55 UTC
We don't have a fix for the root issue - that is being discussed in https://bugzilla.redhat.com/show_bug.cgi?id=2064800. This BZ was mainly for tracking the additional debug data that was dumped.

Comment 8 Priti Kumari 2022-04-11 10:21:45 UTC
As this is additional debug statement. Original issue is yet to address

Verified

Comment 11 errata-xmlrpc 2022-08-10 10:51:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069