Bug 1782982

Summary:

Signatures cannot be verified in airgapped environments or if the remote endpoint goes down

Product:

OpenShift Container Platform

Reporter:

Clayton Coleman <ccoleman>

Component:

Cluster Version Operator

Assignee:

Jack Ottofaro <jack.ottofaro>

Status:

CLOSED ERRATA

QA Contact:

Johnny Liu <jialiu>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

4.3.0

CC:

aos-bugs, aprajapa, asakala, jack.ottofaro, jbasquil, jialiu, jokerman, mhrivnak, susuresh, szobair, thamilto, wking, yanyang

Target Milestone:

---

Target Release:

4.4.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Enhancement

Doc Text:

CVO will now first check for signature configmap on cluster.

Story Points:

---

Clone Of:

Clones:

1783054 1805172 (view as bug list)

Environment:

Last Closed:

2020-05-13 21:54:59 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1805172

Bug Blocks:

1783054

Attachments:

Description	Flags
cvo log	none
cvo log - 1	none

Description Clayton Coleman 2019-12-12 18:35:08 UTC

Currently clusters verify signatures before proceeding with an upgrade.  However, in an airgapped environment or if the upstream endpoint goes down the cluster would be unable to start or restart the upgrade process.

The CVO should:

1. Cache recently verified signatures as long as the payload doesn't change to avoid transient failures
2. Keep an on cluster cache of verified signatures for the current release and any others that may be relevant for use across upgrades
3. Allow an admin to create or update that config map manually

Second tier should haves:

1. CVO should report as a field on available updates whether it is verified
2. We should add oc adm release mirror support for getting the config map of signatures
3. oc adm release info should be able to verify signatures online

Comment 1 W. Trevor King 2020-01-20 22:26:03 UTC

The CVO side of this was:

https://github.com/openshift/cluster-version-operator/pull/279
https://github.com/openshift/cluster-version-operator/pull/303

Not sure if that's sufficient for the bug, or if we're also waiting on getting those moved into library-go and then having 'oc adm release mirror ...' or some such get updated to create the ConfigMap manifests...  But they're enough for signature caching on the CVO side already.

Comment 2 Jack Ottofaro 2020-01-23 17:17:12 UTC

Additional CVO refactor PR and PR to push manifest portion to library-go:

https://github.com/openshift/cluster-version-operator/pull/309
https://github.com/openshift/library-go/pull/680

Comment 3 W. Trevor King 2020-02-13 19:08:35 UTC

Moving this to MODIFIED based on the CVO PRs.  Should get swept into ON_QA when the next nightly is built.  QE will probably need docs around generating signature ConfigMaps manually to test; I'll try to get an openshift-docs PR up soon.

Comment 6 Johnny Liu 2020-02-17 08:27:23 UTC

When you get doc PR ready, pls let me know, so that I can run this bug's verification.

Comment 7 W. Trevor King 2020-02-18 20:50:21 UTC

Once we get swept back into ON_QA, testing will look something like this:

From [1], 4.3.0 digest is sha256:3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d.  So we can get signatures from [2].  Base 64 encoded:

$ ALGO=sha256
$ HASH=3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d
$ curl -s "https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${ALGO}=${HASH}/signature-1" | base64 -w0 && echo
owGbwMvMwMEoOU9/4l9n2UDGtYwySWLxRQW5xZnpukWphbrmvsHOln56SZl5cernJKqVkosySzKTE3OUrBSqlTJzE9NTwayU/OTs1CLd3MS8zLTU4hLdlMx0IAWUUirOSDQyNbMyTjQ1NDOxMEhJSzGzSDVIszBPMzcwSjJJMU9KSTFLM0s0SExOS0lMNk1OMUq1NDczT7IwtkhOTU1JNQaqMUhRqtVRUCqpLABZp5RYkp+bmayQnJ9XkpiZl1qkAHRtXmJJaVGqElBVZkpqXklmSSWyw4pS01KLUvOSwdoLSxMr9TLz9fMLUvOKMzLTSoDSOamJxam6Kall+vnJBTC+lYmesZ6BboWFWbyZiVJtbW0nkygzKwMoJOAhxiEqyf8/2IkhzG/e/j7bmTX95hJ1osltKVVHufInhDBu2lhTMKdKRHXqs6ZN8x3UE+5ODay2b7/x1e38XhHd0FALn4vf2tWiXshb5XjVaqV3FqjYN27ftEH+f52WZzp3upH2l4f3zfq6yjw/s0Y4yqUfd2LOz2d0zFz6uHy/os7vZTPLprxsqNhTxsmcbpIw3Vc0J9XD9MTVzLtclQEeyS5bIkuvWWTVH3j9QKWb+Ue98u1lPBqt35Wrv/bHKrZc7j2Z3jzlqvjWzI57U3nZbbZvPD03RLZgs/pi5ZDrcf5tHberWz4IMnZuZLrnF6Kft7ag9Mb1QF3DiZNaTTfe3NS+8vXTCnfVG49+6+4Wu5XGH5w9c/6M162zMs5LNO7Z7BRSpXWnNK7C6pTUF6/Cn+0/zY/tEPthvzDT+mLpLDunMO/n/FMbXDKWsrdFqjS+Xz55ictOjuNu1gz3+idsDZjgXaoVbd/ao3OS45pvScb7jYXfg/+3XfC+nurwKuRGiJCiVskamd/eRroZPyxUWZy/zj70Q62NX+HnltCz+l/neUwr9VpY+Vv2TKPEp5lPJyxha1iT9Gjq2gkv1Er01wSY6livFQudvUU+kWuS4P2yFbXtt4+fdX6w8njol9BWpYTlBWrJl5csdTmU81O7O5lja5Gz75JPDf1bRASuasiElOdYRB5sW7MxctrSaawpfyNeLHZjmfb44q27Ex8AAA==

ConfigMap will look like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: whatever-but-release-version-or-digest-is-probably-a-good-choice
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
spec:
  binaryData:
    sha256-3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d-1: owGbwMvMwMEoOU9/4l9n2UDGtYwySWLxRQW5xZnpukWphbrmvsHOln56SZl5cernJKqVkosySzKTE3OUrBSqlTJzE9NTwayU/OTs1CLd3MS8zLTU4hLdlMx0IAWUUirOSDQyNbMyTjQ1NDOxMEhJSzGzSDVIszBPMzcwSjJJMU9KSTFLM0s0SExOS0lMNk1OMUq1NDczT7IwtkhOTU1JNQaqMUhRqtVRUCqpLABZp5RYkp+bmayQnJ9XkpiZl1qkAHRtXmJJaVGqElBVZkpqXklmSSWyw4pS01KLUvOSwdoLSxMr9TLz9fMLUvOKMzLTSoDSOamJxam6Kall+vnJBTC+lYmesZ6BboWFWbyZiVJtbW0nkygzKwMoJOAhxiEqyf8/2IkhzG/e/j7bmTX95hJ1osltKVVHufInhDBu2lhTMKdKRHXqs6ZN8x3UE+5ODay2b7/x1e38XhHd0FALn4vf2tWiXshb5XjVaqV3FqjYN27ftEH+f52WZzp3upH2l4f3zfq6yjw/s0Y4yqUfd2LOz2d0zFz6uHy/os7vZTPLprxsqNhTxsmcbpIw3Vc0J9XD9MTVzLtclQEeyS5bIkuvWWTVH3j9QKWb+Ue98u1lPBqt35Wrv/bHKrZc7j2Z3jzlqvjWzI57U3nZbbZvPD03RLZgs/pi5ZDrcf5tHberWz4IMnZuZLrnF6Kft7ag9Mb1QF3DiZNaTTfe3NS+8vXTCnfVG49+6+4Wu5XGH5w9c/6M162zMs5LNO7Z7BRSpXWnNK7C6pTUF6/Cn+0/zY/tEPthvzDT+mLpLDunMO/n/FMbXDKWsrdFqjS+Xz55ictOjuNu1gz3+idsDZjgXaoVbd/ao3OS45pvScb7jYXfg/+3XfC+nurwKuRGiJCiVskamd/eRroZPyxUWZy/zj70Q62NX+HnltCz+l/neUwr9VpY+Vv2TKPEp5lPJyxha1iT9Gjq2gkv1Er01wSY6livFQudvUU+kWuS4P2yFbXtt4+fdX6w8njol9BWpYTlBWrJl5csdTmU81O7O5lja5Gz75JPDf1bRASuasiElOdYRB5sW7MxctrSaawpfyNeLHZjmfb44q27Ex8AAA==

Apply that to your cluster.  Break the CVO's access to mirror.openshift.com and storage.googleapis.com so the CVO can't fetch the signature directly.  Run:

  $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d

Note the lack of --force, because we want to make sure the CVO checks the 4.3.0 signature, which it must get from the local ConfigMap because we broke the HTTPS sig-fetching routes.  4.3.0 CVO should start up and attempt to move the cluster back to 4.3.0.  For the purpose of this bug, we don't care if we finish applying the 4.3.0 manifests.  It is enough to validate the signature and start attempting to apply them.  Workable test plan, jialiu?

[1]: https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.3.0/release.txt
[2]: https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d/

Comment 9 Johnny Liu 2020-02-19 12:56:57 UTC

Retest this bug with 4.4.0-0.nightly-2020-02-18-233330, failed.


1. set up a cluster in a restricted network env after 4.4.0-0.nightly-2020-02-18-233330 is mirrored.
2. mirror 4.4.0-0.nightly-2020-02-19-044512 to local registry.
3. follow the steps in comment 7 to generate signature configmap for 4.4.0-0.nightly-2020-02-19-044512, and create it.
# cat cvo_sig_ConfigMap.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: 94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
spec:
  binaryData:
    sha256-94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1-1: owGbwMvMwME4uavh1CdRx9eMaxkdksTiiwpyizPTdYtSC3VdkyPCc/L0kjLz4nx+s1QrJRdllmQmJ+YoWSlUK2XmJqanglkp+cnZqUW6uYl5mWmpxSW6KZnpQAoopVSckWhkamZlaZKckmaakmaeamZoZGmcZm5gkpiSbJ5immhplGJmlJaUaplsZpmakmiSDFRoYGlmapFiYm5ubmpqaZhskGSoVKujoFRSWQCyTimxJD83M1khOT+vJDEzL7VIAejavMSS0qJUJaCqzJTUvJLMkkpkhxWlpqUWpeYlg7UXliZW6mXm6+cXpOYVZ2SmlQClc1ITi1N1U1LL9POTC+D8vMz0jJKcSisTPRM9A10DPShf18jACMg10jW01DUwMTE1NNKtsDCLNzNRqq2t7WQSZWZlAIUWPFQ5vjHx/w8u3MRyT35r7v8tHpfS97fuvhUlquG9tvDBpHVc7j/9vk9UmRQxe3Hr+8er2eXe/LGcV7Q19dXjY+rx03jq1t6a1/S7WDHq/cnPV+fFHmfJ2PLPt/pb0j6t4O4rZ94qKgSsUBTlCz0YWqYWzrP/fYBx7dGKI7f/rax8pntSOvWDdoYh87vCH5OfZEn4Tt6yyDGpc3nssf8u/3dM/HXl0sZN5/d1h0m3/jh5hefof91NtvI9Wy7eVpURYn4wJ/SXwIp9yxs+FftZPpc+HXRa/PoCrvZnJ3z36lWYbp6tKLnR4MuWXaH685e33eEp1FBWns60UHPNyX3atao/POcxFLrXp8puKortsukxTtc1N2EuPrrQYItbcsf2x+JKm/V+RX9VD/hecm/RJhbBSSsrLovInq1WLHjHpRHRVJ78947GZambpwotV17rvPO88Tq/IUOl0sX7J+WXJSz4ufRfzvq40qDWnpdPlj/8Ut/kYNUW1GLBOy1n62rXv4x5PQ9XcgSoOmcvCH2o4GIZadASllmjNN/g48yPUXfltW3nhSiHRqXxWNXvtK5O81sWz2yd2LuwYiWX793l4tvbzu3jjD7QyvGOgfvUl1nr+GeXL1Nc4W1pYbmf75/fsRTzny6Hj81VrJirbCSlbvZY42PinLbgVww7rjm5qkxS5lp9XSMj3NRI8pzs7Tj5tCen9uw89KxYIE1GUu9EYcwZXwA=

# oc create -f cvo_sig_ConfigMap.yaml 
configmap/94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1 created

# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          3s      Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512: the image may not be safe to use


Go back to check created configmap, seem like no data in itself.
# oc get cm -n openshift-config-managed 94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1 -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: "2020-02-19T12:05:35Z"
  name: 94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1
  namespace: openshift-config-managed
  resourceVersion: "171082"
  selfLink: /api/v1/namespaces/openshift-config-managed/configmaps/94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1
  uid: 651053d5-a773-4ad3-b112-f989311acf7c


Is my configmap yaml file not valid or something missed? Then I tried to using 4.3.0 configmap yaml file in comment 7 to create it. The same behavior.
# cat cvo1.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: jialiu-testing
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
spec:
  binaryData:
    sha256-3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d-1: owGbwMvMwMEoOU9/4l9n2UDGtYwySWLxRQW5xZnpukWphbrmvsHOln56SZl5cernJKqVkosySzKTE3OUrBSqlTJzE9NTwayU/OTs1CLd3MS8zLTU4hLdlMx0IAWUUirOSDQyNbMyTjQ1NDOxMEhJSzGzSDVIszBPMzcwSjJJMU9KSTFLM0s0SExOS0lMNk1OMUq1NDczT7IwtkhOTU1JNQaqMUhRqtVRUCqpLABZp5RYkp+bmayQnJ9XkpiZl1qkAHRtXmJJaVGqElBVZkpqXklmSSWyw4pS01KLUvOSwdoLSxMr9TLz9fMLUvOKMzLTSoDSOamJxam6Kall+vnJBTC+lYmesZ6BboWFWbyZiVJtbW0nkygzKwMoJOAhxiEqyf8/2IkhzG/e/j7bmTX95hJ1osltKVVHufInhDBu2lhTMKdKRHXqs6ZN8x3UE+5ODay2b7/x1e38XhHd0FALn4vf2tWiXshb5XjVaqV3FqjYN27ftEH+f52WZzp3upH2l4f3zfq6yjw/s0Y4yqUfd2LOz2d0zFz6uHy/os7vZTPLprxsqNhTxsmcbpIw3Vc0J9XD9MTVzLtclQEeyS5bIkuvWWTVH3j9QKWb+Ue98u1lPBqt35Wrv/bHKrZc7j2Z3jzlqvjWzI57U3nZbbZvPD03RLZgs/pi5ZDrcf5tHberWz4IMnZuZLrnF6Kft7ag9Mb1QF3DiZNaTTfe3NS+8vXTCnfVG49+6+4Wu5XGH5w9c/6M162zMs5LNO7Z7BRSpXWnNK7C6pTUF6/Cn+0/zY/tEPthvzDT+mLpLDunMO/n/FMbXDKWsrdFqjS+Xz55ictOjuNu1gz3+idsDZjgXaoVbd/ao3OS45pvScb7jYXfg/+3XfC+nurwKuRGiJCiVskamd/eRroZPyxUWZy/zj70Q62NX+HnltCz+l/neUwr9VpY+Vv2TKPEp5lPJyxha1iT9Gjq2gkv1Er01wSY6livFQudvUU+kWuS4P2yFbXtt4+fdX6w8njol9BWpYTlBWrJl5csdTmU81O7O5lja5Gz75JPDf1bRASuasiElOdYRB5sW7MxctrSaawpfyNeLHZjmfb44q27Ex8AAA==

# oc create -f cvo1.yaml
configmap/jialiu-testing created

# oc get cm jialiu-testing  -n openshift-config-managed
NAME             DATA   AGE
jialiu-testing   0      13s


# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          10s     Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64: the image may not be safe to use

Retore, try again.
# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-18-233330 --force
# oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release:4.3.0-x86_64
Updating to release image quay.io/openshift-release-dev/ocp-release:4.3.0-x86_64
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          19s     Unable to apply quay.io/openshift-release-dev/ocp-release:4.3.0-x86_64: the image may not be safe to use

Comment 10 W. Trevor King 2020-02-19 20:19:45 UTC

> Go back to check created configmap, seem like no data in itself.

Oops, looks like ConfigMaps have no spec.  Try with:

apiVersion: v1
kind: ConfigMap
metadata:
  name: 94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
binaryData:
  sha256-94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1-1: owGbwMvMwME4uavh1CdRx9eMaxkdksTiiwpyizPTdYtSC3VdkyPCc/L0kjLz4nx+s1QrJRdllmQmJ+YoWSlUK2XmJqanglkp+cnZqUW6uYl5mWmpxSW6KZnpQAoopVSckWhkamZlaZKckmaakmaeamZoZGmcZm5gkpiSbJ5immhplGJmlJaUaplsZpmakmiSDFRoYGlmapFiYm5ubmpqaZhskGSoVKujoFRSWQCyTimxJD83M1khOT+vJDEzL7VIAejavMSS0qJUJaCqzJTUvJLMkkpkhxWlpqUWpeYlg7UXliZW6mXm6+cXpOYVZ2SmlQClc1ITi1N1U1LL9POTC+D8vMz0jJKcSisTPRM9A10DPShf18jACMg10jW01DUwMTE1NNKtsDCLNzNRqq2t7WQSZWZlAIUWPFQ5vjHx/w8u3MRyT35r7v8tHpfS97fuvhUlquG9tvDBpHVc7j/9vk9UmRQxe3Hr+8er2eXe/LGcV7Q19dXjY+rx03jq1t6a1/S7WDHq/cnPV+fFHmfJ2PLPt/pb0j6t4O4rZ94qKgSsUBTlCz0YWqYWzrP/fYBx7dGKI7f/rax8pntSOvWDdoYh87vCH5OfZEn4Tt6yyDGpc3nssf8u/3dM/HXl0sZN5/d1h0m3/jh5hefof91NtvI9Wy7eVpURYn4wJ/SXwIp9yxs+FftZPpc+HXRa/PoCrvZnJ3z36lWYbp6tKLnR4MuWXaH685e33eEp1FBWns60UHPNyX3atao/POcxFLrXp8puKortsukxTtc1N2EuPrrQYItbcsf2x+JKm/V+RX9VD/hecm/RJhbBSSsrLovInq1WLHjHpRHRVJ78947GZambpwotV17rvPO88Tq/IUOl0sX7J+WXJSz4ufRfzvq40qDWnpdPlj/8Ut/kYNUW1GLBOy1n62rXv4x5PQ9XcgSoOmcvCH2o4GIZadASllmjNN/g48yPUXfltW3nhSiHRqXxWNXvtK5O81sWz2yd2LuwYiWX793l4tvbzu3jjD7QyvGOgfvUl1nr+GeXL1Nc4W1pYbmf75/fsRTzny6Hj81VrJirbCSlbvZY42PinLbgVww7rjm5qkxS5lp9XSMj3NRI8pzs7Tj5tCen9uw89KxYIE1GUu9EYcwZXwA=

Comment 11 Johnny Liu 2020-02-20 02:13:00 UTC

Seem like still not works.

[root@preserve-jialiu-ansible ~]# cat cvo_sig_ConfigMap.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: 94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
binaryData:
  sha256-94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1-1: owGbwMvMwME4uavh1CdRx9eMaxkdksTiiwpyizPTdYtSC3VdkyPCc/L0kjLz4nx+s1QrJRdllmQmJ+YoWSlUK2XmJqanglkp+cnZqUW6uYl5mWmpxSW6KZnpQAoopVSckWhkamZlaZKckmaakmaeamZoZGmcZm5gkpiSbJ5immhplGJmlJaUaplsZpmakmiSDFRoYGlmapFiYm5ubmpqaZhskGSoVKujoFRSWQCyTimxJD83M1khOT+vJDEzL7VIAejavMSS0qJUJaCqzJTUvJLMkkpkhxWlpqUWpeYlg7UXliZW6mXm6+cXpOYVZ2SmlQClc1ITi1N1U1LL9POTC+D8vMz0jJKcSisTPRM9A10DPShf18jACMg10jW01DUwMTE1NNKtsDCLNzNRqq2t7WQSZWZlAIUWPFQ5vjHx/w8u3MRyT35r7v8tHpfS97fuvhUlquG9tvDBpHVc7j/9vk9UmRQxe3Hr+8er2eXe/LGcV7Q19dXjY+rx03jq1t6a1/S7WDHq/cnPV+fFHmfJ2PLPt/pb0j6t4O4rZ94qKgSsUBTlCz0YWqYWzrP/fYBx7dGKI7f/rax8pntSOvWDdoYh87vCH5OfZEn4Tt6yyDGpc3nssf8u/3dM/HXl0sZN5/d1h0m3/jh5hefof91NtvI9Wy7eVpURYn4wJ/SXwIp9yxs+FftZPpc+HXRa/PoCrvZnJ3z36lWYbp6tKLnR4MuWXaH685e33eEp1FBWns60UHPNyX3atao/POcxFLrXp8puKortsukxTtc1N2EuPrrQYItbcsf2x+JKm/V+RX9VD/hecm/RJhbBSSsrLovInq1WLHjHpRHRVJ78947GZambpwotV17rvPO88Tq/IUOl0sX7J+WXJSz4ufRfzvq40qDWnpdPlj/8Ut/kYNUW1GLBOy1n62rXv4x5PQ9XcgSoOmcvCH2o4GIZadASllmjNN/g48yPUXfltW3nhSiHRqXxWNXvtK5O81sWz2yd2LuwYiWX793l4tvbzu3jjD7QyvGOgfvUl1nr+GeXL1Nc4W1pYbmf75/fsRTzny6Hj81VrJirbCSlbvZY42PinLbgVww7rjm5qkxS5lp9XSMj3NRI8pzs7Tj5tCen9uw89KxYIE1GUu9EYcwZXwA=

[root@preserve-jialiu-ansible ~]# oc create -f cvo_sig_ConfigMap.yaml
configmap/94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1 created

[root@preserve-jialiu-ansible ~]# oc get configmap/94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1 -n openshift-config-managed
NAME                                                               DATA   AGE
94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1   1      17s

[root@preserve-jialiu-ansible ~]# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        False         117s    Cluster version is 4.4.0-0.nightly-2020-02-18-233330
[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          0s      Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512: the image may not be safe to use


[root@preserve-jialiu-ansible ~]# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-18-233330 --force
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-18-233330

[root@preserve-jialiu-ansible ~]# cat cvo1.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: jialiu-testing
  namespace: openshift-config-managed
  label:
  - release.openshift.io/verification-signatures
binaryData:
  sha256-3a516480dfd68e0f87f702b4d7bdd6f6a0acfdac5cd2e9767b838ceede34d70d-1: owGbwMvMwMEoOU9/4l9n2UDGtYwySWLxRQW5xZnpukWphbrmvsHOln56SZl5cernJKqVkosySzKTE3OUrBSqlTJzE9NTwayU/OTs1CLd3MS8zLTU4hLdlMx0IAWUUirOSDQyNbMyTjQ1NDOxMEhJSzGzSDVIszBPMzcwSjJJMU9KSTFLM0s0SExOS0lMNk1OMUq1NDczT7IwtkhOTU1JNQaqMUhRqtVRUCqpLABZp5RYkp+bmayQnJ9XkpiZl1qkAHRtXmJJaVGqElBVZkpqXklmSSWyw4pS01KLUvOSwdoLSxMr9TLz9fMLUvOKMzLTSoDSOamJxam6Kall+vnJBTC+lYmesZ6BboWFWbyZiVJtbW0nkygzKwMoJOAhxiEqyf8/2IkhzG/e/j7bmTX95hJ1osltKVVHufInhDBu2lhTMKdKRHXqs6ZN8x3UE+5ODay2b7/x1e38XhHd0FALn4vf2tWiXshb5XjVaqV3FqjYN27ftEH+f52WZzp3upH2l4f3zfq6yjw/s0Y4yqUfd2LOz2d0zFz6uHy/os7vZTPLprxsqNhTxsmcbpIw3Vc0J9XD9MTVzLtclQEeyS5bIkuvWWTVH3j9QKWb+Ue98u1lPBqt35Wrv/bHKrZc7j2Z3jzlqvjWzI57U3nZbbZvPD03RLZgs/pi5ZDrcf5tHberWz4IMnZuZLrnF6Kft7ag9Mb1QF3DiZNaTTfe3NS+8vXTCnfVG49+6+4Wu5XGH5w9c/6M162zMs5LNO7Z7BRSpXWnNK7C6pTUF6/Cn+0/zY/tEPthvzDT+mLpLDunMO/n/FMbXDKWsrdFqjS+Xz55ictOjuNu1gz3+idsDZjgXaoVbd/ao3OS45pvScb7jYXfg/+3XfC+nurwKuRGiJCiVskamd/eRroZPyxUWZy/zj70Q62NX+HnltCz+l/neUwr9VpY+Vv2TKPEp5lPJyxha1iT9Gjq2gkv1Er01wSY6livFQudvUU+kWuS4P2yFbXtt4+fdX6w8njol9BWpYTlBWrJl5csdTmU81O7O5lja5Gz75JPDf1bRASuasiElOdYRB5sW7MxctrSaawpfyNeLHZjmfb44q27Ex8AAA== 

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        False         21s     Cluster version is 4.4.0-0.nightly-2020-02-18-233330

[root@preserve-jialiu-ansible ~]# oc create -f cvo1.yaml
configmap/jialiu-testing created

[root@preserve-jialiu-ansible ~]# oc get cm jialiu-testing -n openshift-config-managed
NAME             DATA   AGE
jialiu-testing   1      53s

[root@preserve-jialiu-ansible ~]# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          2s      Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.3.0-x86_64: the image may not be safe to use

Comment 12 W. Trevor King 2020-02-20 08:39:12 UTC

Can you attach the CVO pod's logs (in the openshift-cluster-version namespace)?  It should include at least one line like:

  An image was retrieved from ... that failed verification: ...

which goes into more detail about why verification failed.

Comment 13 Johnny Liu 2020-02-20 10:51:36 UTC

Created attachment 1664311 [details]
cvo log

Comment 14 Johnny Liu 2020-02-20 10:52:56 UTC

CVO log is attached, go through it, I did not find out "An image was retrieved from ... that failed verification: ...". :-(

Comment 15 W. Trevor King 2020-02-20 10:59:54 UTC

I0220 10:37:57.084414       1 cvo.go:468] Desired version from spec is v1.Update{Version:"", Image:"upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512", Force:false}
I0220 10:37:57.084726       1 sync_worker.go:471] Running sync upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release:4.4.0-0.nightly-2020-02-19-044512 (force=false) on generation 2 in state Updating at attempt 0
I0220 10:37:57.084789       1 sync_worker.go:477] Loading payload
I0220 10:37:57.084831       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (938.01µs)
E0220 10:37:57.084936       1 sync_worker.go:329] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: release images that are not accessed via digest cannot be verified

Can you try again but use a by-digest pullspec instead of the 4.4.0-0.nightly-2020-02-19-044512 tag?  I'll see what we can do about bubbling these messages up into ClusterVersion to make them more discoverable.

Comment 16 Johnny Liu 2020-02-20 12:22:18 UTC

Rested, still fail.

[root@preserve-jialiu-ansible ~]# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-18-233330   True        True          36m     Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1: the image may not be safe to use

[root@preserve-jialiu-ansible ~]# oc logs cluster-version-operator-5799cc5c7f-5rlbd -n openshift-cluster-version>cvo1.log

From log, i see the following:
I0220 11:54:37.364622       1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-02-20 11:54:37.364619164 +0000 UTC m=+1733.643989446)
I0220 11:54:37.364702       1 cvo.go:468] Desired version from spec is v1.Update{Version:"", Image:"upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1", Force:false}
I0220 11:54:37.364933       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (307.465µs)
I0220 11:54:39.617912       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=94cdf5df7e61293f704adc7d5a92d62fbe9c69eda4ccdf09658d47775591c0b1/signature-3: context deadline exceeded
E0220 11:54:39.618097       1 sync_worker.go:329] unable to synchronize image (waiting 2m52.525702462s): The update cannot be verified: context deadline exceeded

This cluster is disconnected cluster, unable to download any file from external site.

Comment 17 Johnny Liu 2020-02-20 12:23:16 UTC

Created attachment 1664341 [details]
cvo log - 1

Comment 18 W. Trevor King 2020-02-20 12:25:35 UTC

> This cluster is disconnected cluster, unable to download any file from external site.

Yup, that's what we want.  I'll drop some logging in around the ConfigMap loading and we'll take another pass once that lands to see why it's not working here.  Thanks for sticking with me on this :).

Comment 19 W. Trevor King 2020-02-20 12:30:31 UTC

I've cloned 1805172 to track the new-in-4.5 changes.  I guess we'll continue there, and then bring them back to this bug and 4.4 once we get it verified.

Comment 20 Jack Ottofaro 2020-02-27 01:07:03 UTC

@jialiu can you test this again but with two changes:

1) For the ConfigMap, be sure to use 'labels' rather than 'label'. My testing showed that 'label' is simply ignored and therefore the CVO will not find the ConfigMap since its searching for label release.openshift.io/verification-signatures as in:

  labels:
    release.openshift.io/verification-signatures: ""

2) Run the upgrade against a regular release rather than a nightly build since the upgrade is to a regular release. The public keys differ.

Here's what I get in CVO log when upgrading the latest 4.4.x regular release which was successful:

I0227 00:56:03.611623       1 cvo.go:254] Verifying release authenticity: All release image digests must have GPG signatures from verifier-public-key-redhat (567E347AD0044ADE55BA8A5F199E2F91FD431D51: Red Hat, Inc. (release key 2) <security>, B08B659EE86AF623BC90E8DB938A80CAF21541EB: Red Hat, Inc. (beta key 2) <security>) - will check for signatures in containers/image format at https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release

However when I tried to upgrade a 4.5-nightly build it was unsuccessful and I got the following:

I0226 20:43:07.206493       1 cvo.go:264] Verifying release authenticity: All release image digests must have GPG signatures from verifier-public-key-openshift-ci (D04761B116203B0C0859B61628B76E05B923888E: openshift-ci) - will check for signatures in containers/image format at https://storage.googleapis.com/openshift-release/test-1/signatures/openshift/release and from config maps in openshift-config-managed with label "release.openshift.io/verification-signatures"

Thanks.

Comment 21 Johnny Liu 2020-02-27 02:41:39 UTC

(In reply to Jack Ottofaro from comment #20)
> 
> 2) Run the upgrade against a regular release rather than a nightly build
> since the upgrade is to a regular release. The public keys differ.
Do you mean "a regular release" is released version signed with release key?
Can you point me the exact version for "4.4.x regular release" in your testing?
If you are saying about a released stable build, I can try that. But for QE's 
regular testing, that sound like unreasonable. For QE, we need run our testing 
using pre-release/nightly build to find out issue prior to its release. QE 
can not bear such risk of pending our testing until build is totally released out.
And in my above testing, I already co-worked with W. Trevor King and confirmed
my selected 4.4 nightly build is already signed with beta2 key.

Comment 22 Jack Ottofaro 2020-02-27 03:59:03 UTC

(In reply to Johnny Liu from comment #21)
> (In reply to Jack Ottofaro from comment #20)
> > 
> > 2) Run the upgrade against a regular release rather than a nightly build
> > since the upgrade is to a regular release. The public keys differ.
> Do you mean "a regular release" is released version signed with release key?
> Can you point me the exact version for "4.4.x regular release" in your
> testing?
> If you are saying about a released stable build, I can try that. But for
> QE's 
> regular testing, that sound like unreasonable. For QE, we need run our
> testing 
> using pre-release/nightly build to find out issue prior to its release. QE 
> can not bear such risk of pending our testing until build is totally
> released out.
> And in my above testing, I already co-worked with W. Trevor King and
> confirmed
> my selected 4.4 nightly build is already signed with beta2 key.

Can you try your same test again with just the change to 'labels' -> change 1) I mentioned.

Comment 23 Johnny Liu 2020-02-27 04:20:32 UTC

Yeah, in progress. Once have result, will post back here.

Comment 24 Jack Ottofaro 2020-02-27 05:04:27 UTC

(In reply to Johnny Liu from comment #23)
> Yeah, in progress. Once have result, will post back here.

Thanks. I'm not sure about #2 but will check in the morning here in EST. However, going with my theory, you could test with a nightly by attempting to upgrade to another nightly. I know the signature you're using went with the 4.3 "upgrade to" release but I think it's pulling the public key to do the verification from the current release so we have verifier-public-key-openshift-ci for nightly's and verifier-public-key-redhat for the others.

Comment 27 Johnny Liu 2020-03-02 10:35:09 UTC

For openshift cluster-version-operator pull 332, run the following steps to verify on 4.4.0-0.nightly-2020-02-29-235938.

Create signature config map file for 4.4.0-0.nightly-2020-03-01-215047, remove label for this configmap.

# oc get cm e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d -o yaml -n openshift-config-managed
apiVersion: v1
binaryData:
  sha256-e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d-1: owGbwMvMwME4uavh1CdRx9eMaxkdksTiiwpyizPTdYtSC3UtC3MKUhL1kjLz4mI6JlcrJRdllmQmJ+YoWSlUK2XmJqanglkp+cnZqUW6uYl5mWmpxSW6KZnpQAoopVSckWhkamaVap5oaGGRampqkpyUaGaWaplqmJSSaJ5obJmYbGFgaGhoYWqSaGZoZmSUZGFgZG5uYmaZlGZgYZxmapqaZGqWolSro6BUUlkAsk4psSQ/NzNZITk/ryQxMy+1SAHo2rzEktKiVCWgqsyU1LySzJJKZIcVpaalFqXmJYO1F5YmVupl5uvnF6TmFWdkppUApXNSE4tTdVNSy/Tzkwvg/LzM9IySnEorEz0TPQNdAz0oX9fIwAjINdY1MNQ1MjQ1MDHXrbAwizczUaqtre1kEmVmZQCFFjxUOTzn8v8v27I32XxZ+SuL9MZALfaZUbm6Xv4s6gLJu684dzg+vLsje5GSYsGCJOa7tSJ1Mff+vl8tZ7hwM1flav/ImvTGvQd4w5T8ZQ87hvbEa23rvS3IcOHOoZsTkx5YRJi2/FFu+ZlRPznf6M/3STztGc76X2+svj/lzjSZ+ObPZ4rs2z7+Zrqlnq58P3O1ZK0lp6er3ZbHDR+2Gn3kUchPX3C//muQk0tBl3D9+Yv8M5+kxM/fePbmHCV9+yONi48wauT9/m/+eKVulJtZbdvk18vFl6j/upNefdjYcm2KxLMnXiyP8pSuL21+W7Rp+bOpGSlFAuvebvnn+kUw53FT1xQHi+fNdoyvlYo7oiP5nph+TxOf1MI4f8J/abZMp5RcTtfCRbcWv9Wddpr5zZcDX3KYpDwPtF0oDo42+TYh9775xj5zzROxvwM+PXn7V+zBTJvwY9sUI6SFjrbp7s0u5tl6p7TI8OTPOzLzVVbL5OVnePktP91UdcJf8WxRa1pEpU3BjHVbk+NLggJ5HWUTXLlfSzxNXdkbpmrawWL34X5Mmt1fFTHXtXOygzNK/2jLnNvjFVZlbvqU+6Xa58Uty9fLlX6aeC1ayOyqwfL/xpIMtc/yUgr1WDY+4VNjTo1ZIKiZuNLg6FmuV9V8Dk2f3t9dG2Oi3X9Ia+VzkbXLul46syi79/Zph2eFzp763i82cH9ep0EBvzp/ojUA
kind: ConfigMap
metadata:
  creationTimestamp: "2020-03-02T08:40:40Z"
  name: e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d
  namespace: openshift-config-managed
  resourceVersion: "133711"
  selfLink: /api/v1/namespaces/openshift-config-managed/configmaps/e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d
  uid: d4403355-8561-4614-a23d-35b981ed943f
 
Trigger an upgrade toward 4.4.0-0.nightly-2020-03-01-215047.
[root@preserve-jialiu-ansible ~]# oc adm upgrade --allow-explicit-upgrade --to-image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d
Updating to release image upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-02-29-235938   True        True          5m59s   Unable to apply upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d: the image may not be safe to use

Capture cvo log, found the following lines:
5113 I0302 10:26:56.465586       1 store.go:74] use cached most recent signature config maps
5114 I0302 10:26:56.464818       1 cvo.go:503] Started syncing available updates "openshift-cluster-version/version" (2020-03-02 10:26:56.464807818 +0000 UTC m=+1042.124957534)
5115 I0302 10:26:56.478991       1 store.go:65] remember most recent signature config maps: signatures-managed
5116 I0302 10:26:56.479094       1 store.go:116] searching for sha256-e7a188e554cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d in signature config map signatures-managed
5117 I0302 10:26:59.857229       1 reflector.go:241] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: forcing resync
5118 I0302 10:26:59.857416       1 cvo.go:526] Started syncing upgradeable "openshift-cluster-version/version" (2020-03-02 10:26:59.857411452 +0000 UTC m=+1045.517561141)
5119 I0302 10:26:59.857480       1 upgradeable.go:28] Upgradeable conditions were recently checked, will try later.
5120 I0302 10:26:59.857501       1 cvo.go:528] Finished syncing upgradeable "openshift-cluster-version/version" (87.413µs)
5121 I0302 10:26:59.857481       1 cvo.go:439] Started syncing cluster version "openshift-cluster-version/version" (2020-03-02 10:26:59.857469169 +0000 UTC m=+1045.517618898)
5122 I0302 10:26:59.857546       1 cvo.go:468] Desired version from spec is v1.Update{Version:"", Image:"upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release@sha256:e7a188e55     4cba66e9e1bda7a39ac80111854a61622b80277469bf083f55eb56d", Force:false}
<--snip-->
5159 I0302 10:29:06.405840       1 cvo.go:441] Finished syncing cluster version "openshift-cluster-version/version" (186.417µs)
5160 I0302 10:29:07.377364       1 verify.go:404] unable to load signature: Get https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=e7a188e554cba66e     9e1bda7a39ac80111854a61622b80277469bf083f55eb56d/signature-1: dial tcp 172.217.9.208:443: connect: connection timed out

The log is already printed in CVO log.

Comment 34 errata-xmlrpc 2020-05-13 21:54:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581