1878925 – 'oc adm upgrade --to ...' rejects versions which occur only in history, while the cluster-version operator supports history fallback

Bug 1878925 - 'oc adm upgrade --to ...' rejects versions which occur only in history, while the cluster-version operator supports history fallback

Summary: 'oc adm upgrade --to ...' rejects versions which occur only in history, while...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	oc
Sub Component:
Version:	4.1.z
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	low
Target Milestone:	---
Target Release:	4.10.0
Assignee:	W. Trevor King
QA Contact:	zhou ying
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-14 22:59 UTC by W. Trevor King
Modified:	2022-03-11 18:15 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-03-11 18:15:11 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-version-operator pull 684	0	None	Merged	Bug 1878925: pkg/cvo/updatepayload: Drop history from target pullspec lookup	2022-03-10 23:37:25 UTC
Github	openshift oc pull 566	0	None	Merged	Bug 1878925: pkg/cli/admin/upgrade: Remove help text around history lookups	2022-03-10 23:37:24 UTC

Description W. Trevor King 2020-09-14 22:59:15 UTC

Description of problem:

'oc adm upgrade --to ...' does not consider ClusterVersion's status.history when attempting to construct an update from a historical version name.

Version-Release number of selected component (if applicable):

oc has never supported this, while the CVO has fallen back to history since 2018 [1].

How reproducible: 100%

Steps to Reproduce:
1. Install a cluster with release A.
2. Update to release B.
3. Use 'oc adm upgrade --to A' to attempt a return to A.

Actual results:

error: Can't look up image for version 4.6.0-fc.5. The update channel has not been configured.

The error message could also be:

* No available updates, specify --to-image or wait for new updates to be available
* The update A is not one of the available updates: ...

Expected results:

oc updates spec.desiredUpdate and we proceed to update to A. 

Additional info:

You can work around this by skipping 'oc adm upgrade' and patching ClusterVersion yourself:

  $ oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/desiredUpdate", "value": {"version": "4.6.0-fc.5"}}]'

[1]: https://github.com/openshift/cluster-version-operator/commit/558ae021bd79a781a9398e6c9ba9a0a9cdf52c92#diff-d540a41404f678a1c438f4c1e5b92a87R332

Comment 2 W. Trevor King 2020-12-04 22:40:36 UTC

I haven't been able to drive review on the PR to a conclusion.

Comment 5 W. Trevor King 2021-11-05 15:27:59 UTC

Ok, both PR's landed.  The consensus direction was "we don't want to support history lookups", so verification with the new oc should look like:

  $ oc adm upgrade --help | grep " --to='"
        --to='': Specify the version to upgrade to. The version must be on the list of available updates.

(where before this bug, it also included "to previous"), and:

  $ oc adm upgrade --allow-explicit-upgrade --to $SOME_VERSION_STRING_FROM_YOUR_HISTORY

(where before this bug, the CVO accepted the update, and with this bug, the CVO complains about not being able to find a pullspec for that not-recommended-for-your-cluster-now version).

Comment 8 Yang Yang 2021-11-11 10:05:55 UTC

Reproduced with Client Version: 4.9.0-0.nightly-2021-09-21-215600

# oc adm upgrade --help | grep " --to='"
      --to='': Specify the version to upgrade to. The version must be on the list of previous or available updates.

# oc adm upgrade channel candidate-4.9
warning: No channels known to be compatible with the current version "4.8.19"; unable to validate "candidate-4.9". Setting the update channel to "candidate-4.9" anyway.

# oc -n openshift-config patch cm admin-acks --patch '{"data":{"ack-4.8-kube-1.22-api-removals-in-4.9":"true"}}' --type=merge
configmap/admin-acks patched

# oc adm upgrade --to=4.9.6
Updating to 4.9.6

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.6     True        False         14m     Cluster version is 4.9.6

# oc get clusterversion/version -ojson | jq -r .status.history
[
  {
    "completionTime": "2021-11-11T09:42:46Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:c9f58ccb8a9085df4eeb23e21ca201d4c7d39bc434786d58a55381e13215a199",
    "startedTime": "2021-11-11T08:31:45Z",
    "state": "Completed",
    "verified": true,
    "version": "4.9.6"
  },
  {
    "completionTime": "2021-11-11T07:24:04Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59",
    "startedTime": "2021-11-11T07:00:10Z",
    "state": "Completed",
    "verified": false,
    "version": "4.8.19"
  }
]

# oc adm upgrade 
Cluster version is 4.9.6

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.9 (available channels: candidate-4.9, fast-4.9)
Updates:

VERSION IMAGE
4.9.7   quay.io/openshift-release-dev/ocp-release@sha256:5c55be02e32e688ec5a404858a08cf533ba15b50b6f0e028089635b47db5866e


# oc adm upgrade --to 4.8.19
error: The update 4.8.19 is not one of the available updates: 4.9.7

# oc adm upgrade --to 4.8.19 --allow-explicit-upgrade
error: The update 4.8.19 is not one of the available updates: 4.9.7

Comment 9 Yang Yang 2021-11-12 06:13:22 UTC

Continue to reproduce it:

Patch the desired version to a historic version
# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.6     True        False         9m48s   Cluster version is 4.9.6

# oc adm upgrade 
Cluster version is 4.9.6

Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.9 (available channels: candidate-4.9, fast-4.9)
Updates:

VERSION IMAGE
4.9.7   quay.io/openshift-release-dev/ocp-release@sha256:5c55be02e32e688ec5a404858a08cf533ba15b50b6f0e028089635b47db5866e

# oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/desiredUpdate", "value": {"version": "4.8.19"}}]'
clusterversion.config.openshift.io/version patched

# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.6     True        True          89s     Working towards 4.8.19: 71 of 681 done (10% complete)

# oc get clusterversion/version -ojson | jq -r .status.history
[
  {
    "completionTime": null,
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59",
    "startedTime": "2021-11-12T06:07:59Z",
    "state": "Partial",
    "verified": true,
    "version": "4.8.19"
  },
  {
    "completionTime": "2021-11-12T05:47:42Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:c9f58ccb8a9085df4eeb23e21ca201d4c7d39bc434786d58a55381e13215a199",
    "startedTime": "2021-11-12T04:40:51Z",
    "state": "Completed",
    "verified": true,
    "version": "4.9.6"
  },
  {
    "completionTime": "2021-11-12T03:44:51Z",
    "image": "quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59",
    "startedTime": "2021-11-12T03:21:33Z",
    "state": "Completed",
    "verified": false,
    "version": "4.8.19"
  }
]

CVO allows to update to a historic version.

Comment 10 Yang Yang 2021-11-12 06:27:03 UTC

Verifying it with 

# ./oc version
Client Version: 4.10.0-0.nightly-2021-11-09-181140
Server Version: 4.10.0-0.nightly-2021-11-09-181140
Kubernetes Version: v1.22.1+1b2affc

Verifying oc change:
# ./oc adm upgrade --help | grep " --to='"
      --to='': Specify the version to upgrade to. The version must be on the list of available updates.

oc change is verified passed.


Verifying cvo change:
# oc patch clusterversion/version --patch '{"spec":{"upstream":"https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph"}}' --type=merge
clusterversion.config.openshift.io/version patched

# ./oc adm upgrade channel nightly-4.10
warning: No channels known to be compatible with the current version "4.10.0-0.nightly-2021-11-09-181140"; unable to validate "nightly-4.10". Setting the update channel to "nightly-4.10" anyway.

# ./oc adm upgrade 
Cluster version is 4.10.0-0.nightly-2021-11-09-181140

Upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Channel: nightly-4.10
Available Updates:

VERSION                            IMAGE
4.10.0-0.nightly-2021-11-10-212548 registry.ci.openshift.org/ocp/release@sha256:b15acfa35c303c15148e1032774c91df0b38ea2b3efee4d8c408777d64467c70
4.10.0-0.nightly-2021-11-11-072405 registry.ci.openshift.org/ocp/release@sha256:4a4004a27b74f1f9a229755d9cb77701823ddbba4377bf090a1bfa4579e80d37
4.10.0-0.nightly-2021-11-11-170956 registry.ci.openshift.org/ocp/release@sha256:3b5eeefd3ba57ae2ffe81b34516ab7330fe966067a5ca467fb40d9476905b400
4.10.0-0.nightly-2021-11-12-023027 registry.ci.openshift.org/ocp/release@sha256:7635f6abdcff00ea285d8f85a4cacafed564bd9c9ecbe783cdc3afbc746f1b89

# ./oc adm upgrade --to 4.10.0-0.nightly-2021-11-11-072405 --force
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to 4.10.0-0.nightly-2021-11-11-072405

# ./oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-11-11-072405   True        False         7m7s    Cluster version is 4.10.0-0.nightly-2021-11-11-072405

# ./oc adm upgrade
Cluster version is 4.10.0-0.nightly-2021-11-11-072405

Upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Channel: nightly-4.10
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.


Patch to a historic version:
# ./oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/desiredUpdate", "value": {"version": "4.10.0-0.nightly-2021-11-09-181140"}}]'
clusterversion.config.openshift.io/version patched

# ./oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-11-11-072405   True        False         14m     Cluster version is 4.10.0-0.nightly-2021-11-11-072405

./oc get clusterversion -oyaml
clusterversion.config.openshift.io/version patched
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
    creationTimestamp: "2021-11-12T02:03:54Z"
    generation: 14
    name: version
    resourceVersion: "111774"
    uid: e6d25362-c3a5-4e20-8db3-881fd81eab04
  spec:
    channel: nightly-4.10
    clusterID: 5b61a0c2-3a48-4dd2-93e9-92a0cc099e0e
    desiredUpdate:
      version: 4.10.0-0.nightly-2021-11-09-181140
    upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
  status:
    availableUpdates: null
    conditions:
    - lastTransitionTime: "2021-11-12T02:27:44Z"
      message: Done applying 4.10.0-0.nightly-2021-11-11-072405
      status: "True"
      type: Available
    - lastTransitionTime: "2021-11-12T05:16:52Z"
      status: "False"
      type: Failing
    - lastTransitionTime: "2021-11-12T06:05:31Z"
      message: Cluster version is 4.10.0-0.nightly-2021-11-11-072405
      status: "False"
      type: Progressing
    - lastTransitionTime: "2021-11-12T02:37:40Z"
      status: "True"
      type: RetrievedUpdates
    desired:
      image: registry.ci.openshift.org/ocp/release@sha256:4a4004a27b74f1f9a229755d9cb77701823ddbba4377bf090a1bfa4579e80d37
      version: 4.10.0-0.nightly-2021-11-11-072405
    history:
    - completionTime: "2021-11-12T05:50:16Z"
      image: registry.ci.openshift.org/ocp/release@sha256:4a4004a27b74f1f9a229755d9cb77701823ddbba4377bf090a1bfa4579e80d37
      startedTime: "2021-11-12T04:58:14Z"
      state: Completed
      verified: false
      version: 4.10.0-0.nightly-2021-11-11-072405
    - completionTime: "2021-11-12T04:56:59Z"
      image: registry.ci.openshift.org/ocp/release@sha256:ab7bf0469aa054834c7c43a717a53b4903ee854dc4894bf92c816c28d2d1c05a
      startedTime: "2021-11-12T04:56:29Z"
      state: Completed
      verified: false
      version: 4.10.0-0.nightly-2021-11-09-181140
    - completionTime: "2021-11-12T04:56:29Z"
      image: registry.ci.openshift.org/ocp/release@sha256:4a4004a27b74f1f9a229755d9cb77701823ddbba4377bf090a1bfa4579e80d37
      startedTime: "2021-11-12T04:55:44Z"
      state: Partial
      verified: false
      version: 4.10.0-0.nightly-2021-11-11-072405
    - completionTime: "2021-11-12T02:27:44Z"
      image: registry.ci.openshift.org/ocp/release@sha256:ab7bf0469aa054834c7c43a717a53b4903ee854dc4894bf92c816c28d2d1c05a
      startedTime: "2021-11-12T02:03:54Z"
      state: Completed
      verified: false
      version: 4.10.0-0.nightly-2021-11-09-181140
    observedGeneration: 13
    versionHash: eOxGyGcqLo4=
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Update to a historic version not started which is as expected.

Trevor, CVO does not prompt any messages to indicate why the rollback does not get started. Do you think it needs an enhancement?

Comment 11 Yang Yang 2021-11-17 06:07:16 UTC

Moving it to verified because oc and CVO have the consistent behavior on rolling back to historical version.

Comment 12 W. Trevor King 2021-12-14 00:23:30 UTC

An enhancement around CVO reporting that it cannot find a pullspec for the requested release version makes sense to me.  Possibly as part of bug 1822752, or a follow up to it, see discussion in [1].

[1]: https://github.com/openshift/cluster-version-operator/pull/683#discussion_r767446179

Comment 13 Yang Yang 2021-12-14 02:09:39 UTC

Thanks Trevor! I'll come back to it to see what will happen after Bug 1822752 gets fixed.

Note You need to log in before you can comment on or make changes to this bug.