Bug 2062568
Summary: | CVO does not trigger new upgrade again after fail to update to unavailable payload | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | liujia <jiajliu> | |
Component: | Cluster Version Operator | Assignee: | W. Trevor King <wking> | |
Status: | CLOSED ERRATA | QA Contact: | liujia <jiajliu> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 4.11 | CC: | asheth, wking | |
Target Milestone: | --- | |||
Target Release: | 4.11.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | If docs needed, set a value | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 2071211 (view as bug list) | Environment: | ||
Last Closed: | 2022-08-10 10:53:13 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 2071211 |
Description
liujia
2022-03-10 07:14:35 UTC
Not a blocker. Most release-pulling jobs will succeed in less than the job timeout, which is two minutes [1]. I suspect you'd need to connect to a very slow registry or have other pod-launching issues to hit this. Mitigation should be possible via: $ oc -n openshift-cluster-version delete jobs --all or similar to remove the failed job. Inspecting the failed job to understand why it failed would also be useful. [1]: https://github.com/openshift/cluster-version-operator/blob/0e9bc4ef03004fcc2bb0c58d39e5b49445a9f8f6/pkg/cvo/updatepayload.go#L162 Version: 4.11.0-0.nightly-2022-03-13-055724 1. Upgrade from 4.11.0-0.nightly-2022-03-13-055724 to 4.11.0-0.nightly-2022-03-14-113722 with wrong repo # ./oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:f21d4bb9ccb8a82cc14906bf89b0422ffd5c423b5e5dfc10b843957181de87f2 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:f21d4bb9ccb8a82cc14906bf89b0422ffd5c423b5e5dfc10b843957181de87f2 2. No upgrade happen # ./oc get clusterversion -ojson|jq .items[].status.conditions ... { "lastTransitionTime": "2022-03-15T01:20:26Z", "message": "Retrieving payload failed version=\"\" image=\"quay.io/openshift-release-dev/ocp-release@sha256:f21d4bb9ccb8a82cc14906bf89b0422ffd5c423b5e5dfc10b843957181de87f2\" failure=Unable to download and prepare the update: deadline exceeded, reason: \"DeadlineExceeded\", message: \"Job was active longer than specified deadline\"", "reason": "RetrievePayload", "status": "False", "type": "ReleaseAccepted" }, ... # ./oc get clusterversion -ojson|jq .items[].status.history [ { "completionTime": "2022-03-15T00:59:59Z", "image": "registry.ci.openshift.org/ocp/release@sha256:9653c71def3a3cf89e2b973a0328ac684f8bb6f913eab3bfbd106737fe09e57c", "startedTime": "2022-03-15T00:42:38Z", "state": "Completed", "verified": false, "version": "4.11.0-0.nightly-2022-03-13-055724" } ] 3. Continue upgrade to target payload with correct repo. # ./oc adm upgrade --to-image registry.ci.openshift.org/ocp/release@sha256:f21d4bb9ccb8a82cc14906bf89b0422ffd5c423b5e5dfc10b843957181de87f2 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway Updating to release image registry.ci.openshift.org/ocp/release@sha256:f21d4bb9ccb8a82cc14906bf89b0422ffd5c423b5e5dfc10b843957181de87f2 4. Upgrade is triggered successfully. # ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-03-13-055724 True True 58s Working towards 4.11.0-0.nightly-2022-03-14-113722: 118 of 777 done (15% complete) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |