Bug 1840343

Summary: Parallelize external signature fetching to avoid a slow store denying access to subsequent stores
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: Cluster Version OperatorAssignee: W. Trevor King <wking>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.zCC: aos-bugs, jokerman, rh-container
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: HTTPS signature retrieval serialized stores. Consequence: A slow primary signature store could cause overall retrieval to time out before the cluster-version operator attempted to retrieve signatures from later stores. Fix: External, HTTPS signature retrieval is now parallel. Result: All stores will be attempted, regardless of the latency of any single store.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:01:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1857478, 1906498    

Description W. Trevor King 2020-05-26 18:38:49 UTC 
From [1]:

$ grep 'unable to load signature' cvo.log 
I0526 13:13:36.123153       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-1: dial tcp 172.217.4.48:443: connect: connection timed out
I0526 13:15:47.195128       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-2: dial tcp 172.217.9.80:443: connect: connection timed out
I0526 13:17:10.718027       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-3: context deadline exceeded
I0526 13:19:44.764143       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-1: dial tcp 172.217.4.48:443: connect: connection timed out
I0526 13:21:55.835063       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-2: dial tcp 172.217.4.48:443: connect: connection timed out
I0526 13:23:18.233801       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=baa687f29b0ac155d8f4c6914056d36d68f343feb9c1e82b46eef95819d00be5/signature-3: context deadline exceeded

In this case, that was because a restricted network blocked access to storage.googleapis.com, but you'd get similar behavior if storage.googleapis.com itself was slow.  We want to hit other signature sources (like our mirrors [2]) before giving up on signatures entirely, and a robust way to do that would be to walk all external signature stores in parallel.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1838497#c10
[2]: https://github.com/openshift/cluster-update-keys/blob/cca4ce696383e70ae669e770bd63265a9540b721/manifests.rhel/0000_90_cluster-update-keys_configmap.yaml#L5
Comment 1 Lalatendu Mohanty 2020-05-28 10:49:02 UTC 
We are working on feature development and bugs which are of higher priority. Hence moving this to the new sprint.
Comment 2 Lalatendu Mohanty 2020-06-18 14:45:41 UTC 
Hitting storage.googleapis.com and mirror.openhshift.com one after another if one is not available or in case of error is a better idea.
Comment 7 liujia 2020-07-13 09:09:55 UTC 
Verified on 4.6.0-0.nightly-2020-07-13-071854

# ./oc logs cluster-version-operator-6d46b5b676-fvqmz |grep 'unable to load'|sort -u
I0713 08:59:38.677664       1 sigstore.go:98] unable to load signature: Get "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=3168403d8bae362beaf7708b17ddeecb84e421e80620a59aa7c5e1e6731b2ea4/signature-1": Forbidden
I0713 08:59:38.677898       1 sigstore.go:98] unable to load signature: Get "https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=3168403d8bae362beaf7708b17ddeecb84e421e80620a59aa7c5e1e6731b2ea4/signature-1": Forbidden
I0713 08:59:38.678869       1 sigstore.go:98] unable to load signature: Get "https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=3168403d8bae362beaf7708b17ddeecb84e421e80620a59aa7c5e1e6731b2ea4/signature-2": Forbidden
I0713 08:59:38.679036       1 sigstore.go:98] unable to load signature: Get "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=3168403d8bae362beaf7708b17ddeecb84e421e80620a59aa7c5e1e6731b2ea4/signature-2": Forbidden
...
Comment 9 errata-xmlrpc 2020-10-27 16:01:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196