Description of problem: When user accepts the risk and goes ahead with a conditional edge, it should populate the history with acceptedRisks on Recommended=False updates Version-Release number of the following components: How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated Expected results: Additional info: Please attach logs from ansible-playbook with the -vvv flag
I'd like to see this fixed with some sense of urgency. We're particularly blind on what risks people are acknowledging when they take an upgrade with risks.
Hi Jack, Does force upgrade have possibility to have the AcceptedRisks in history? I did a 4.12 to 4.12 force upgrade since no signed/accepted nightlies recently. But AcceptedRisks was not in the history. # oc adm upgrade --include-not-recommended Cluster version is 4.12.0-0.nightly-2022-10-13-083325 Upstream: https://storage.googleapis.com/ota-cincy/OCP-47176_200985-cincy-20221014-023340.json Channel: stable-4.12 No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss. Supported but not recommended updates: Version: 4.12.0-0.nightly-2022-10-13-123950 Image: registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1 Recommended: False Reason: SomeInvokerThing Message: On clusters on default invoker user, this imaginary bug can happen. https://bug.example.com/a Version: 4.0.0 Image: 1111111111111111111111111111111111111111111111111111111111111111 Recommended: False Reason: SomeInvokerThing Message: On clusters on default invoker user, this imaginary bug can happen. https://bug.example.com/a 4.12.0-0.nightly-2022-10-13-123950 has Recommended: False Lets update to 4.12.0-0.nightly-2022-10-13-123950 # oc adm upgrade info: An upgrade is in progress. Working towards 4.12.0-0.nightly-2022-10-13-123950: 129 of 831 done (15% complete), waiting on kube-controller-manager Upstream: https://storage.googleapis.com/ota-cincy/OCP-47176_200985-cincy-20221014-023340.json Channel: stable-4.12 No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss. Fine, upgrade is in progress # grep -i recommendedupdate cvo1 I1014 06:37:18.188406 1 status.go:88] merge into existing history completed=false desired=v1.Release{Version:"4.12.0-0.nightly-2022-10-13-123950", Image:"registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1", URL:"", Channels:[]string(nil)} last=&v1.UpdateHistory{State:"Partial", StartedTime:time.Date(2022, time.October, 14, 6, 37, 14, 0, time.Local), CompletionTime:<nil>, Version:"4.12.0-0.nightly-2022-10-13-123950", Image:"registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1", Verified:false, AcceptedRisks:"Target release version=\"4.12.0-0.nightly-2022-10-13-123950\" image=\"registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1\" cannot be verified, but continuing anyway because the update was forced: unable to verify sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1 against keyrings: verifier-public-key-redhat\n[2022-10-14T06:37:00Z: prefix sha256-291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1 in config map signatures-managed: no more signatures to check, 2022-10-14T06:37:00Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1/signature-1: no more signatures to check, 2022-10-14T06:37:01Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1/signature-1: no more signatures to check, 2022-10-14T06:37:01Z: parallel signature store wrapping containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release, containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release: no more signatures to check, 2022-10-14T06:37:01Z: serial signature store wrapping config maps in openshift-config-managed with label \"release.openshift.io/verification-signatures\", parallel signature store wrapping containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release, containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release: no more signatures to check]\nForced through blocking failures: Precondition \"ClusterVersionRecommendedUpdate\" failed because of \"SomeInvokerThing\": Update from 4.12.0-0.nightly-2022-10-13-083325 to 4.12.0-0.nightly-2022-10-13-123950 is not recommended:\n\nOn clusters on default invoker user, this imaginary bug can happen. https://bug.example.com/a"} We can see the Precondition ClusterVersionRecommendedUpdate failed in the cvo log. But we don't see the AcceptedRisks in the history # oc get clusterversion/version -ojson | jq -r .status.history [ { "completionTime": "2022-10-14T07:15:03Z", "image": "registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1", "startedTime": "2022-10-14T06:37:14Z", "state": "Completed", "verified": false, "version": "4.12.0-0.nightly-2022-10-13-123950" }, { "completionTime": "2022-10-14T06:23:34Z", "image": "registry.ci.openshift.org/ocp/release@sha256:6a892b5b5b0c0ce1d99df3edc0835388056b5bbaabb5bd0cdb7fccfb0959aea8", "startedTime": "2022-10-14T06:00:56Z", "state": "Completed", "verified": false, "version": "4.12.0-0.nightly-2022-10-13-083325" } ] # grep AcceptedRisks cvo1 | tail -1 I1014 07:23:48.189142 1 status.go:88] merge into existing history completed=true desired=v1.Release{Version:"4.12.0-0.nightly-2022-10-13-123950", Image:"registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1", URL:"", Channels:[]string(nil)} last=&v1.UpdateHistory{State:"Completed", StartedTime:time.Date(2022, time.October, 14, 6, 37, 14, 0, time.Local), CompletionTime:time.Date(2022, time.October, 14, 7, 15, 3, 0, time.Local), Version:"4.12.0-0.nightly-2022-10-13-123950", Image:"registry.ci.openshift.org/ocp/release@sha256:291d0a7e13038c3b8b810cd4bf5e4329210817c867151e6d2170d1f3ab5050b1", Verified:false, AcceptedRisks:""}
Per the description of acceptedRisks acceptedRisks: description: acceptedRisks records risks which were accepted to initiate the update. For example, it may menition an Upgradeable=False or missing signature that was overriden via desiredUpdate.force, or an update that was initiated despite not being in the availableUpdates set of recommended update targets. sounds like force upgrade and recommended=false would be recorded in my testing. But I don't see the acceptedRisks entry in comment#4. Re-opening it.
(In reply to Yang Yang from comment #7) > Per the description of acceptedRisks > > acceptedRisks: > description: acceptedRisks records risks which were > accepted > to initiate the update. For example, it may menition > an Upgradeable=False > or missing signature that was overriden via > desiredUpdate.force, > or an update that was initiated despite not being in > the availableUpdates > set of recommended update targets. > > sounds like force upgrade and recommended=false would be recorded in my > testing. But I don't see the acceptedRisks entry in comment#4. Re-opening it. Yes, forcing past verification should show up in history's accepted risks. What's happening is when CVO restarts and loads the payload on init, verification is not done again, so history's accepted risks for the payload gets clobbered. Thanks for finding. I'll send up a fix.
Verifying on 4.12.0-0.nightly-2022-10-22-063823. We got the acceptedRisks finally and it complained about force upgrade, the image signature and recommended=false upgrade. But the signature message looks too long and the format is not that good. Jack, it's functional so I'd like to close it. Would you mind me opening a bug against the message? # oc get clusterversion -oyaml ...... history: - acceptedRisks: |- Target release version="4.12.0-0.nightly-2022-10-23-154914" image="registry.ci.openshift.org/ocp/release@sha256:03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366" cannot be verified, but continuing anyway because the update was forced: unable to verify sha256:03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 against keyrings: verifier-public-key-redhat [2022-10-24T02:22:58Z: prefix sha256-03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 in config map signatures-managed: no more signatures to check, 2022-10-24T02:22:58Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366/signature-1: no more signatures to check, 2022-10-24T02:22:58Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366/signature-1: no more signatures to check, 2022-10-24T02:22:58Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2022-10-24T02:22:58Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check] Forced through blocking failures: Precondition "ClusterVersionRecommendedUpdate" failed because of "SomeInvokerThing": Update from 4.12.0-0.nightly-2022-10-22-063823 to 4.12.0-0.nightly-2022-10-23-154914 is not recommended: On clusters on default invoker user, this imaginary bug can happen. https://bug.example.com/a completionTime: "2022-10-24T02:58:15Z" image: registry.ci.openshift.org/ocp/release@sha256:03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 startedTime: "2022-10-24T02:23:11Z" state: Completed verified: false version: 4.12.0-0.nightly-2022-10-23-154914
(In reply to Yang Yang from comment #11) > Verifying on 4.12.0-0.nightly-2022-10-22-063823. > > We got the acceptedRisks finally and it complained about force upgrade, the > image signature and recommended=false upgrade. But the signature message > looks too long and the format is not that good. > > Jack, it's functional so I'd like to close it. Would you mind me opening a > bug against the message? > > # oc get clusterversion -oyaml > ...... > history: > - acceptedRisks: |- > Target release version="4.12.0-0.nightly-2022-10-23-154914" > image="registry.ci.openshift.org/ocp/release@sha256: > 03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366" cannot be > verified, but continuing anyway because the update was forced: unable to > verify > sha256:03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 > against keyrings: verifier-public-key-redhat > [2022-10-24T02:22:58Z: prefix > sha256-03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 in > config map signatures-managed: no more signatures to check, > 2022-10-24T02:22:58Z: unable to retrieve signature from > https://storage.googleapis.com/openshift-release/official/signatures/ > openshift/release/ > sha256=03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366/ > signature-1: no more signatures to check, 2022-10-24T02:22:58Z: unable to > retrieve signature from > https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/ > sha256=03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366/ > signature-1: no more signatures to check, 2022-10-24T02:22:58Z: parallel > signature store wrapping containers/image signature store under > https://storage.googleapis.com/openshift-release/official/signatures/ > openshift/release, containers/image signature store under > https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: > no more signatures to check, 2022-10-24T02:22:58Z: serial signature store > wrapping config maps in openshift-config-managed with label > "release.openshift.io/verification-signatures", parallel signature store > wrapping containers/image signature store under > https://storage.googleapis.com/openshift-release/official/signatures/ > openshift/release, containers/image signature store under > https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: > no more signatures to check] > Forced through blocking failures: Precondition > "ClusterVersionRecommendedUpdate" failed because of "SomeInvokerThing": > Update from 4.12.0-0.nightly-2022-10-22-063823 to > 4.12.0-0.nightly-2022-10-23-154914 is not recommended: > > On clusters on default invoker user, this imaginary bug can happen. > https://bug.example.com/a > completionTime: "2022-10-24T02:58:15Z" > image: > registry.ci.openshift.org/ocp/release@sha256: > 03a3d3288070259e8cd3e60b3ca7da1c8d410fb3620c44f2e1422dd4fb8f4366 > startedTime: "2022-10-24T02:23:11Z" > state: Completed > verified: false > version: 4.12.0-0.nightly-2022-10-23-154914 Sure, that's fine.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399