Bug 2064800

Summary: OpenStack proxy job permafailing with: Image signature workflow can push a signed image to openshift registry and verify it
Product: OpenShift Container Platform Reporter: Adam Kaplan <adam.kaplan>
Component: Image RegistryAssignee: Flavian <fmissi>
Status: CLOSED ERRATA QA Contact: XiuJuan Wang <xiuwang>
Severity: medium Docs Contact:
Priority: high    
Version: 4.9CC: jitsingh, m.andre, obulatov, pbhattac, pprinett, spandura
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2057990 Environment:
Last Closed: 2022-06-30 05:31:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2061554    
Bug Blocks:    

Description Adam Kaplan 2022-03-16 15:06:35 UTC
+++ This bug was initially created as a clone of Bug #2057990 +++

OpenShift-on-OpenStack 4.9 Proxy tests are permafailing[1]. The  test consistently failing is: "[sig-imageregistry][Serial][Suite:openshift/registry/serial] Image signature workflow can push a signed image to openshift registry and verify it [Suite:openshift/conformance/serial]".

The same test is consistently succeeding on OCP v4.10.

This possibly is a follow-up to Bug 2041358

[1]: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.9-e2e-openstack-proxy

Comment 1 Martin André 2022-03-23 13:55:21 UTC
Depends on https://bugzilla.redhat.com/show_bug.cgi?id=2063284 actually (the 4.9 backport), for which https://github.com/openshift/origin/pull/26903 adds the necessary debugging info.

Comment 2 Corey Daley 2022-04-06 13:33:53 UTC
It looks like the pull request which adds the debug information has merged and the extra debug info is showing in the latest runs: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.9-e2e-openstack-proxy/1510698555369590784

Comment 3 Adam Kaplan 2022-04-06 19:12:29 UTC
Moving this to the Image Registry team.

There seem to be a few issues going on here:

1. The image signature workflow test appears to run in 4.9, but is skipped or does not exist at all in 4.10 and above. See https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.10-e2e-openstack-proxy/1510698551959621632

2. The build that starts the whole process installs skopeo from the ubi8 repos. The content is served through the cluster's proxy, and we get the following error:

```
Errors during downloading metadata for repository 'cdn-ubi.redhat.com_content_public_ubi_dist_ubi8_8_basearch_appstream_os_':
  - Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi8/8/x86_64/appstream/os/repodata/repomd.xml [SSL certificate problem: self signed certificate]
Error: Failed to download metadata for repo 'cdn-ubi.redhat.com_content_public_ubi_dist_ubi8_8_basearch_appstream_os_': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
error: build error: error building at STEP "RUN yum install -y skopeo &&     yum clean all && mkdir -p gnupg && chmod -R 0777 /var/lib/origin": error while running runtime: exit status 1
```

It is possible that the test is not applying proxy's CA bundle to the cluster, and thus builds can't trust anything that goes through the proxy.

Comment 11 errata-xmlrpc 2022-06-30 05:31:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.40 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5180