Bug 2075647
| Summary: | 'oc adm upgrade ...' POSTs ClusterVersion, clobbering any unrecognized spec properties | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | oc | Assignee: | W. Trevor King <wking> |
| oc sub component: | oc | QA Contact: | Evgeni Vakhonin <evakhoni> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | high | CC: | aos-bugs, mfojtik, travi, yanyang |
| Version: | 4.1.z | ||
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 11:07:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2077332 | ||
reproduced on Client Version: 4.11.0-0.nightly-2022-04-16-163450
$ oc get clusterversions.config.openshift.io version -ojson | jq -r '"spec", .spec, "status caps", .status.capabilities'
spec
{
"capabilities": {
"additionalEnabledCapabilities": [
"marketplace"
],
"baselineCapabilitySet": "None"
},
"channel": "stable-4.11",
"clusterID": "4da7abd2-2237-4e8b-9a58-ab3ba99d34b6",
"upstream": "https://amd64.ocp.releases.ci.openshift.org/graph"
}
status caps
{
"enabledCapabilities": [
"marketplace"
],
"knownCapabilities": [
"baremetal",
"marketplace",
"openshift-samples"
]
}
$ oc adm upgrade --allow-explicit-upgrade --force --to-image=registry.ci.openshift.org/ocprelease@sha256:96e0eb4aa27b39553ca01a4614105c6e980503ad1cdc584cf7a06a792c831391
$ oc get clusterversions.config.openshift.io version -ojson | jq -r '"spec", .spec, "status caps", .status.capabilities'
spec
{
"channel": "stable-4.11",
"clusterID": "4da7abd2-2237-4e8b-9a58-ab3ba99d34b6",
"desiredUpdate": {
"force": true,
"image": "registry.ci.openshift.org/ocp/release@sha256:96e0eb4aa27b39553ca01a4614105c6e980503ad1cdc584cf7a06a792c831391",
"version": ""
},
"upstream": "https://amd64.ocp.releases.ci.openshift.org/graph"
}
status caps
{
"enabledCapabilities": [
"baremetal",
"marketplace",
"openshift-samples"
],
"knownCapabilities": [
"baremetal",
"marketplace",
"openshift-samples"
]
}
.spec.capabilities dropped by oc client, resulting in unwanted enabledCapabilities
verifying on Client Version: 4.11.0-0.nightly-2022-04-20-002931
$ oc get clusterversions.config.openshift.io version -ojson | jq -r '"spec", .spec, "status caps", .status.capabilities'
spec
{
"capabilities": {
"additionalEnabledCapabilities": [
"marketplace"
],
"baselineCapabilitySet": "None"
},
"channel": "stable-4.11",
"clusterID": "cad19f59-08b8-4391-bf4b-eb071b4cc1f2"
}
status caps
{
"enabledCapabilities": [
"marketplace"
],
"knownCapabilities": [
"baremetal",
"marketplace",
"openshift-samples"
]
}
$ oc adm upgrade --allow-explicit-upgrade --force --to-image=registry.ci.openshift.org/ocprelease@sha256:e385a786f122c6c0e8848ecb9901f510676438f17af8a5c4c206807a9bc0bf28
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image registry.ci.openshift.org/ocprelease@sha256:e385a786f122c6c0e8848ecb9901f510676438f17af8a5c4c206807a9bc0bf28
$ oc get clusterversions.config.openshift.io version -ojson | jq -r '"spec", .spec, "status caps", .status.capabilities'
spec
{
"capabilities": {
"additionalEnabledCapabilities": [
"marketplace"
],
"baselineCapabilitySet": "None"
},
"channel": "stable-4.11",
"clusterID": "cad19f59-08b8-4391-bf4b-eb071b4cc1f2",
"desiredUpdate": {
"force": true,
"image": "registry.ci.openshift.org/ocprelease@sha256:e385a786f122c6c0e8848ecb9901f510676438f17af8a5c4c206807a9bc0bf28",
"version": ""
}
}
status caps
{
"enabledCapabilities": [
"marketplace"
],
"knownCapabilities": [
"baremetal",
"marketplace",
"openshift-samples"
]
}
so far so good! no spec dropped, no unwanted caps installed!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Since before we were releasing 4.y OpenShift,'oc adm upgrade ...' has used Update(...) to set ClusterVersion's spec.desiredVersion [1]. The spec structure had been very stable for years, but added 'capabilities' for 4.11 [2]. That means that older oc, who do not know about the new spec.capabilities property would have flows like: 1. Admin calls 'oc adm upgrade --to ...' or similar update request. 2. oc GETs the ClusterVersion and unpacks it into the vendored Go type [3]. For older oc, where that vendored Go type lacks a new property, the new property is silently dropped at this point. 3. oc fiddles around with the local Go structure, making changes to the properties it does know and care about. 4. oc POSTs ClusterVersion with Update(...) [4]. The Kubernetes API service recives the serialized spec structure, picks up the desired property changes, and also picks up the undesired dropped-in-step-2 property removals. We should be PATCHing in step 4 instead, so that both the client and the service are on the same page about which properties are being touched. We have a few guards on impact: * [5] discusses genetic oc-vs-cluster compatibility, with "oc client might not be able to access server features" when oc is from an older 4.y minor than the service. It also says: Non-security API changes will involve, at minimum, two minor releases (4.1 to 4.2 to 4.3, for example) to allow older oc binaries to update. * Since bug 2067291, [6] has: Update OpenShift CLI oc to the target version before each update. * Since 4.1 [7], [8] has: Install the OpenShift CLI (oc) that matches the version for your updated version. So folks who are reading those docs carefully shouldn't be using 4.10 or earlier oc to request updates to 4.11 or later, and only 4.11 clusters will have the new 'capabilities' field. But that's a leaning fairly heavily on "folks who are reading those docs carefully" and also relies on the update-specific doc sections, so it doesn't protect other oc subcommands that might have similar GET/Go-struct/POST exposure. [1]: https://github.com/openshift/oc/commit/11d6adcaa97180344114d9fe212ddbb6c4511672#diff-65b6199099c673e23589269abae7fddf0d4d849ee63fe6a6d77d458dfe766f59L174 [2]: https://github.com/openshift/api/blame/143629bf6d5fe114a1c1672cdfd07a34168a8d45/config/v1/types_cluster_version.go#L35-L82 [3]: https://github.com/openshift/oc/blob/1b0e16c050128800e6e1f5f1827a875a92c69541/pkg/cli/admin/upgrade/upgrade.go#L170 [4]: https://github.com/openshift/oc/blob/1b0e16c050128800e6e1f5f1827a875a92c69541/pkg/cli/admin/upgrade/upgrade.go#L340 [5]: https://docs.openshift.com/container-platform/4.10/cli_reference/openshift_cli/usage-oc-kubectl.html#the-oc-binary [6]: https://docs.openshift.com/container-platform/4.10/updating/preparing-eus-eus-upgrade.html#updating-eus-to-eus-upgrade_eus-to-eus-upgrade [7]: https://github.com/openshift/openshift-docs/pull/13993/commits/0d4d9888826a91bb36c4a98cf3706b903511a6fa#diff-c53e73b13c36b252211983803b66aa8af998646d08f67ef3ff2f6ae86d1c84faR24 [8]: https://docs.openshift.com/container-platform/4.10/updating/updating-cluster-cli.html#update-upgrading-cli_updating-cluster-cli