Bug 1999092 - [4.8] Should not allow upgrades to 4.9 without admin acknowledgement that apis are being removed
Summary: [4.8] Should not allow upgrades to 4.9 without admin acknowledgement that api...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.9
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.z
Assignee: Jack Ottofaro
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On: 1978376
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-30 12:08 UTC by OpenShift BugZilla Robot
Modified: 2021-10-27 21:33 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-12 06:01:20 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-version-operator pull 647 0 None None None 2021-08-31 14:51:59 UTC
Red Hat Product Errata RHBA-2021:3682 0 None None None 2021-10-12 06:01:45 UTC

Comment 1 Johnny Liu 2021-09-09 08:30:00 UTC
Do we have some document issue or bug to tracking this enhancement?

Comment 2 Jack Ottofaro 2021-09-09 12:50:40 UTC
(In reply to Johnny Liu from comment #1)
> Do we have some document issue or bug to tracking this enhancement?

This was just opened https://issues.redhat.com/browse/OSDOCS-2627

Comment 3 Johnny Liu 2021-09-18 10:19:36 UTC
A potential issue is raised in my mind:

Assume this PR would land into 4.8.z, how to prevent all the 4.8 clusters on < 4.8.z version from being upgraded to 4.9, so that force all 4.8 clusters bumped onto the latest 4.8.z before move forwards to 4.9?

Comment 4 Johnny Liu 2021-09-18 10:20:48 UTC
utilize cincinnati-graph ?

Comment 5 W. Trevor King 2021-09-19 04:10:11 UTC
Exactly.  Once this goes out in a 4.8.z, we'll raise 4.9's minor_min [1] to match.  So the GA and later 4.9 will only have recommended updates from 4.8 releases that include this gate.

[1]: https://github.com/openshift/cincinnati-graph-data/blob/a88269fc8fac9aeefcf1973cc186d25ffd92c33a/build-suggestions/4.9.yaml#L5

Comment 6 W. Trevor King 2021-10-01 20:29:12 UTC
I've created bug 2009879 to track the origin test-case side of this effort, so this one can go to MODIFIED.

Comment 9 Johnny Liu 2021-10-08 12:13:48 UTC
Verified this bug with 4.8.14, and PASS.

1. Install a cluster with 4.8.13, ensure there is no any admin-gates.
$ oc -n openshift-config-managed get configmap admin-gates -o json | jq -r ".data"
Error from server (NotFound): configmaps "admin-gates" not found

2. Upgrade the cluster to 4.8.14, ensure admin-gates is existing.
$ oc -n openshift-config-managed get configmap admin-gates -o json | jq -r ".data"
{
  "ack-4.8-kube-1.22-api-removals-in-4.9": "Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see\nthe knowledge article https://access.redhat.com/articles/6329921 for details and instructions.\n"
}

$ oc adm upgrade
Cluster version is 4.8.14

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.

3. Try to upgrade the cluster to a nightly build
$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:10183ffb82d31f58a9f03ee3de19da1d43a038c495e3d9e0d7781cc9c01d80ea --allow-explicit-upgrade
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image registry.ci.openshift.org/ocp/release@sha256:10183ffb82d31f58a9f03ee3de19da1d43a038c495e3d9e0d7781cc9c01d80ea

$ oc adm upgrade
info: An upgrade is in progress. Unable to apply registry.ci.openshift.org/ocp/release@sha256:10183ffb82d31f58a9f03ee3de19da1d43a038c495e3d9e0d7781cc9c01d80ea: the image may not be safe to use

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.


4. The nightly build is not signed, so the above z-stream upgrade is blocked, cancel the upgrade.
$ oc adm upgrade --clear
Cleared the update field, still at registry.ci.openshift.org/ocp/release@sha256:10183ffb82d31f58a9f03ee3de19da1d43a038c495e3d9e0d7781cc9c01d80ea

$ oc adm upgrade
Cluster version is 4.8.14

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions

5. Without ack admin-gates, try to upgrade the cluster to 4.9.0-rc.5.
$ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:a8c3e839ef81ba46e37f4150b5129e37577844ea1025d4c5703fc97518d30451
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway

Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:a8c3e839ef81ba46e37f4150b5129e37577844ea1025d4c5703fc97518d30451

$ oc adm upgrade
info: An upgrade is in progress. Unable to apply 4.9.0-rc.5: it may not be safe to apply this update

Upgradeable=False

Reason: AdminAckRequired
Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.

6. Roll back, and ack admin-gates
$ oc -n openshift-config patch configmap admin-acks --patch '{"data":{"ack-4.8-kube-1.22-api-removals-in-4.9": "true"}}' --type=merge

7. Trigger upgrade again
$ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:a8c3e839ef81ba46e37f4150b5129e37577844ea1025d4c5703fc97518d30451

warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:a8c3e839ef81ba46e37f4150b5129e37577844ea1025d4c5703fc97518d30451

$ oc adm upgrade
info: An upgrade is in progress. Working towards 4.9.0-rc.5: 71 of 734 done (9% complete)


Upgrade is started.

Comment 11 errata-xmlrpc 2021-10-12 06:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.14 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3682

Comment 12 W. Trevor King 2021-10-27 21:33:38 UTC
Removing the UpgradeBlocker reminder keyword per [1].

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1978376#c29


Note You need to log in before you can comment on or make changes to this bug.