Bug 1795160

Summary:

Upgrade stuck with ImageVerificationFailed

Product:

OpenShift Container Platform

Reporter:

Andre Costa <andcosta>

Component:

Cluster Version Operator

Assignee:

Abhinav Dahiya <adahiya>

Status:

CLOSED DUPLICATE

QA Contact:

liujia <jiajliu>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.2.z

CC:

aos-bugs, jokerman

Target Milestone:

---

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-01-27 14:48:33 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
clusterversion-operator_log	none

Description Andre Costa 2020-01-27 09:53:30 UTC

Description of problem:
Cluster version operator keeps stuck in progressing with ImageVerificationFailed:
I0116 06:25:35.194000       1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-9: dial tcp 172.217.22.80:443: connect: connection refused
I0116 06:25:35.229044       1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release

At first suspected the reason was the proxy settings not be configured, so asked the customer to create the proxy/cluster config, but after that the issue still persists.

Version-Release number of the following components:
OCP 4.2.8
Upgrade to 4.2.13


Steps to Reproduce:
Unknown

clusterversion status:
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2019-12-04T12:31:43Z"
    generation: 2
    name: version
    resourceVersion: "18160575"
    selfLink: /apis/config.openshift.io/v1/clusterversions/version
    uid: 07883a0a-1692-11ea-bc0e-001a4a160334
  spec:
    channel: stable-4.2
    clusterID: 2867b9f6-480e-4df0-aff8-a6f2f030518e
    desiredUpdate:
      force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    upstream: https://api.openshift.com/api/upgrades_info/v1/graph
  status:
    availableUpdates:
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:f28cbabd1227352fe704a00df796a4511880174042dece96233036a10ac61639
      version: 4.2.9
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:3fabe939da31f9a31f509251b9f73d321e367aba2d09ff392c2f452f6433a95a
      version: 4.2.14
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:77ade34c373062c6a6c869e0e56ef93b2faaa373adadaac1430b29484a24d843
      version: 4.2.12
    - force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:dc2e38fb00085d6b7f722475f8b7b758a0cb3a02ba42d9acf8a8298a6d510d9c
      version: 4.2.10
    conditions:
    - lastTransitionTime: "2019-12-05T06:49:37Z"
      message: Done applying 4.2.8
      status: "True"
      type: Available
    - lastTransitionTime: "2020-01-17T09:58:09Z"
      message: 'The update cannot be verified: unable to locate a valid signature
        for one or more sources'
      reason: ImageVerificationFailed
      status: "True"
      type: Failing
    - lastTransitionTime: "2020-01-13T10:37:11Z"
      message: 'Unable to apply 4.2.13: the image may not be safe to use'
      reason: ImageVerificationFailed
      status: "True"
      type: Progressing
    - lastTransitionTime: "2020-01-17T07:52:03Z"
      status: "True"
      type: RetrievedUpdates
    desired:
      force: false
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      version: 4.2.13
    history:
    - completionTime: null
      image: quay.io/openshift-release-dev/ocp-release@sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d
      startedTime: "2020-01-13T10:37:11Z"
      state: Partial
      verified: false
      version: 4.2.13
    - completionTime: "2019-12-05T06:49:37Z"
      image: quay.io/openshift-release-dev/ocp-release@sha256:4bf307b98beba4d42da3316464013eac120c6e5a398646863ef92b0e2c621230
      startedTime: "2019-12-04T12:31:43Z"
      state: Completed
      verified: false
      version: 4.2.8
    observedGeneration: 2
    versionHash: e65s6d-vrKY=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 1 Andre Costa 2020-01-27 09:54:20 UTC

Created attachment 1655640 [details]
clusterversion-operator_log

Comment 2 Scott Dodson 2020-01-27 14:48:33 UTC

It does not appear to be using the proxy, as you can see below it's connecting directly to the signature stores.

I0116 06:30:37.200291       1 cvo.go:354] Finished syncing cluster version "openshift-cluster-version/version" (149.47µs)
I0116 06:30:37.260939       1 verify.go:332] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 172.217.22.80:443: connect: connection refused
...
I0116 06:30:37.645403       1 verify.go:332] unable to load signature: Get https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d/signature-1: dial tcp 54.173.18.88:443: connect: connection refused
I0116 06:30:37.938641       1 verify.go:270] Unable to verify sha256:782b41750f3284f3c8ee2c1f8cb896896da074e362cf8a472846356d1617752d against keyring verifier-public-key-redhat
E0116 06:30:37.938733       1 sync_worker.go:311] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: unable to locate a valid signature for one or more sources

This was fixed in 4.2.13, however that only affects CVO after you've updated to that version so you'll need to work around this by setting HTTP_PROXY, HTTPS_PROXY, on the current CVO deployment. The bug I'm duping this of has several private comments where it's been confirmed that setting those variables unblocks the upgrade.

*** This bug has been marked as a duplicate of bug 1775836 ***