Description of problem: Cluster version operator keeps stuck in progressing with ImageVerificationFailed: I0116 06:25:35.194000 1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-9: dial tcp 172.217.22.80:443: connect: connection refused I0116 06:25:35.229044 1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release At first suspected the reason was the proxy settings not be configured, so asked the customer to create the proxy/cluster config, but after that the issue still persists. Version-Release number of the following components: OCP 4.2.8 Upgrade to 4.2.13 Steps to Reproduce: Unknown clusterversion status: apiVersion: v1 items: - apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: creationTimestamp: "2019-12-04T12:31:43Z" generation: 2 name: version resourceVersion: "18160575" selfLink: /apis/config.openshift.io/v1/clusterversions/version uid: 07883a0a-1692-11ea-bc0e-001a4a160334 spec: channel: stable-4.2 clusterID: 2867b9f6-480e-4df0-aff8-a6f2f030518e desiredUpdate: force: false image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d version: 4.2.13 upstream: https://api.openshift.com/api/upgrades_info/v1/graph status: availableUpdates: - force: false image: quay.io/openshift-release-dev/ocp-release@sha256:f28cbabd1227352fe704a00df796a4511880174042dece96233036a10ac61639 version: 4.2.9 - force: false image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d version: 4.2.13 - force: false image: quay.io/openshift-release-dev/ocp-release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a version: 4.2.14 - force: false image: quay.io/openshift-release-dev/ocp-release@sha256:77ade34c373062c6a6c869e0e56ef93b2faaa373adadaac1430b29484a24d843 version: 4.2.12 - force: false image: quay.io/openshift-release-dev/ocp-release@sha256:dc2e38fb00085d6b7f722475f8b7b758a0cb3a02ba42d9acf8a8298a6d510d9c version: 4.2.10 conditions: - lastTransitionTime: "2019-12-05T06:49:37Z" message: Done applying 4.2.8 status: "True" type: Available - lastTransitionTime: "2020-01-17T09:58:09Z" message: 'The update cannot be verified: unable to locate a valid signature for one or more sources' reason: ImageVerificationFailed status: "True" type: Failing - lastTransitionTime: "2020-01-13T10:37:11Z" message: 'Unable to apply 4.2.13: the image may not be safe to use' reason: ImageVerificationFailed status: "True" type: Progressing - lastTransitionTime: "2020-01-17T07:52:03Z" status: "True" type: RetrievedUpdates desired: force: false image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d version: 4.2.13 history: - completionTime: null image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d startedTime: "2020-01-13T10:37:11Z" state: Partial verified: false version: 4.2.13 - completionTime: "2019-12-05T06:49:37Z" image: quay.io/openshift-release-dev/ocp-release@sha256:4bf307b98beba4d42da3316464013eac120c6e5a398646863ef92b0e2c621230 startedTime: "2019-12-04T12:31:43Z" state: Completed verified: false version: 4.2.8 observedGeneration: 2 versionHash: e65s6d-vrKY= kind: List metadata: resourceVersion: "" selfLink: ""
Created attachment 1655640 [details] clusterversion-operator_log
It does not appear to be using the proxy, as you can see below it's connecting directly to the signature stores. I0116 06:30:37.200291 1 cvo.go:354] Finished syncing cluster version "openshift-cluster-version/version" (149.47µs) I0116 06:30:37.260939 1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 172.217.22.80:443: connect: connection refused ... I0116 06:30:37.645403 1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 54.173.18.88:443: connect: connection refused I0116 06:30:37.938641 1 verify.go:270] Unable to verify sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d against keyring verifier-public-key-redhat E0116 06:30:37.938733 1 sync_worker.go:311] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: unable to locate a valid signature for one or more sources This was fixed in 4.2.13, however that only affects CVO after you've updated to that version so you'll need to work around this by setting HTTP_PROXY, HTTPS_PROXY, on the current CVO deployment. The bug I'm duping this of has several private comments where it's been confirmed that setting those variables unblocks the upgrade. *** This bug has been marked as a duplicate of bug 1775836 ***