Bug 1795160 - Upgrade stuck with ImageVerificationFailed
Summary: Upgrade stuck with ImageVerificationFailed
Keywords:
Status: CLOSED DUPLICATE of bug 1775836
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.2.z
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Abhinav Dahiya
QA Contact: liujia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-27 09:53 UTC by Andre Costa
Modified: 2023-09-07 21:35 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-27 14:48:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
clusterversion-operator_log (82.16 KB, text/plain)
2020-01-27 09:54 UTC, Andre Costa
no flags Details

Description Andre Costa 2020-01-27 09:53:30 UTC
Description of problem:
Cluster version operator keeps stuck in progressing with ImageVerificationFailed:
I0116 06:25:35.194000       1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-9: dial tcp 172.217.22.80:443: connect: connection refused
I0116 06:25:35.229044       1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release

At first suspected the reason was the proxy settings not be configured, so asked the customer to create the proxy/cluster config, but after that the issue still persists.

Version-Release number of the following components:
OCP 4.2.8
Upgrade to 4.2.13


Steps to Reproduce:
Unknown

clusterversion status:
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2019-12-04T12:31:43Z"
    generation: 2
    name: version
    resourceVersion: "18160575"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: 07883a0a-1692-11ea-bc0e-001a4a160334
  spec:
    channel: stable-4.2
    clusterID: 2867b9f6-480e-4df0-aff8-a6f2f030518e
    desiredUpdate:
      force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    upstream: https://api.openshift.com/api/upgrades_info/v1/graph
  status:
    availableUpdates:
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:f28cbabd1227352fe704a00df796a4511880174042dece96233036a10ac61639
      version: 4.2.9
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a
      version: 4.2.14
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:77ade34c373062c6a6c869e0e56ef93b2faaa373adadaac1430b29484a24d843
      version: 4.2.12
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:dc2e38fb00085d6b7f722475f8b7b758a0cb3a02ba42d9acf8a8298a6d510d9c
      version: 4.2.10
    conditions:
    - lastTransitionTime: "2019-12-05T06:49:37Z"
      message: Done applying 4.2.8
      status: "True"
      type: Available
    - lastTransitionTime: "2020-01-17T09:58:09Z"
      message: 'The update cannot be verified: unable to locate a valid signature
        for one or more sources'
      reason: ImageVerificationFailed
      status: "True"
      type: Failing
    - lastTransitionTime: "2020-01-13T10:37:11Z"
      message: 'Unable to apply 4.2.13: the image may not be safe to use'
      reason: ImageVerificationFailed
      status: "True"
      type: Progressing
    - lastTransitionTime: "2020-01-17T07:52:03Z"
      status: "True"
      type: RetrievedUpdates
    desired:
      force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    history:
    - completionTime: null
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      startedTime: "2020-01-13T10:37:11Z"
      state: Partial
      verified: false
      version: 4.2.13
    - completionTime: "2019-12-05T06:49:37Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:4bf307b98beba4d42da3316464013eac120c6e5a398646863ef92b0e2c621230
      startedTime: "2019-12-04T12:31:43Z"
      state: Completed
      verified: false
      version: 4.2.8
    observedGeneration: 2
    versionHash: e65s6d-vrKY=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 1 Andre Costa 2020-01-27 09:54:20 UTC
Created attachment 1655640 [details]
clusterversion-operator_log

Comment 2 Scott Dodson 2020-01-27 14:48:33 UTC
It does not appear to be using the proxy, as you can see below it's connecting directly to the signature stores.

I0116 06:30:37.200291       1 cvo.go:354] Finished syncing cluster version "openshift-cluster-version/version" (149.47µs)
I0116 06:30:37.260939       1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 172.217.22.80:443: connect: connection refused
...
I0116 06:30:37.645403       1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 54.173.18.88:443: connect: connection refused
I0116 06:30:37.938641       1 verify.go:270] Unable to verify sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d against keyring verifier-public-key-redhat
E0116 06:30:37.938733       1 sync_worker.go:311] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: unable to locate a valid signature for one or more sources

This was fixed in 4.2.13, however that only affects CVO after you've updated to that version so you'll need to work around this by setting HTTP_PROXY, HTTPS_PROXY, on the current CVO deployment. The bug I'm duping this of has several private comments where it's been confirmed that setting those variables unblocks the upgrade.

*** This bug has been marked as a duplicate of bug 1775836 ***


Note You need to log in before you can comment on or make changes to this bug.