Bug 1952230
| Summary: | 'oc adm upgrade' only lists one client-side guard failure | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | oc | Assignee: | W. Trevor King <wking> |
| Status: | CLOSED WONTFIX | QA Contact: | Yang Yang <yanyang> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 4.2.0 | CC: | aos-bugs, mfojtik, yanyang |
| Target Milestone: | --- | ||
| Target Release: | 4.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | LifecycleReset | ||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-12 13:14:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1992680 | ||
| Bug Blocks: | |||
|
Description
W. Trevor King
2021-04-21 19:53:27 UTC
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Whiteboard if you think this bug should never be marked as stale. Please consult with bug assignee before you do that. The LifecycleStale keyword was removed because the bug moved to QE. The bug assignee was notified. Reproduced it with Client Version: 4.9.0-0.nightly-2021-09-21-215600:
1. Install a 4.8 cluster
# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.19 True False 20h Cluster version is 4.8.19
2. Make co degraded
cat <<EOF >oauth.yaml
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
name: cluster
spec:
identityProviders:
- name: oidcidp
mappingMethod: claim
type: OpenID
openID:
clientID: test
clientSecret:
name: test
claims:
preferredUsername:
- preferred_username
name:
- name
email:
- email
issuer: https://www.idp-issuer.example.com
EOF
# ./oc apply -f oauth.yaml
oauth.config.openshift.io/cluster configured
Then, we get authentication degraded, mco unavailable
# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.8.19 True False True 4h17m
baremetal 4.8.19 True False False 23h
cloud-credential 4.8.19 True False False 23h
cluster-autoscaler 4.8.19 True False False 23h
config-operator 4.8.19 True False False 23h
console 4.8.19 True False False 22h
csi-snapshot-controller 4.8.19 True False False 23h
dns 4.8.19 True False False 23h
etcd 4.8.19 True False False 23h
image-registry 4.8.19 True False False 22h
ingress 4.8.19 True False False 22h
insights 4.8.19 True False False 22h
kube-apiserver 4.8.19 True False False 23h
kube-controller-manager 4.8.19 True False False 23h
kube-scheduler 4.8.19 True False False 23h
kube-storage-version-migrator 4.8.19 True False False 23h
machine-api 4.8.19 True False False 22h
machine-approver 4.8.19 True False False 23h
machine-config 4.8.19 False False True 35s
marketplace 4.8.19 True False False 23h
monitoring 4.8.19 True False False 22h
network 4.8.19 True False False 23h
node-tuning 4.8.19 True False False 23h
openshift-apiserver 4.8.19 True False False 22h
openshift-controller-manager 4.8.19 True False False 3h56m
openshift-samples 4.8.19 True False False 22h
operator-lifecycle-manager 4.8.19 True False False 23h
operator-lifecycle-manager-catalog 4.8.19 True False False 23h
operator-lifecycle-manager-packageserver 4.8.19 True False False 22h
service-ca 4.8.19 True False False 23h
storage 4.8.19 True False False 23h
3. Patch to update the channel and desired version
# oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "candidate-4.9"}, {"op": "add", "path": "/spec/desiredUpdate", "value": {"version": "4.9.6"}}]'
clusterversion.config.openshift.io/version patched
# oc adm upgrade
Error while reconciling 4.8.19: the cluster operator authentication is degraded
Upgradeable=False
Reason: AdminAckRequired
Message: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.
Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.9
warning: Cannot display available updates:
Reason: VersionNotFound
Message: Unable to retrieve available updates: currently reconciling cluster version 4.8.19 not found in the "fast-4.9" channel
So, oc adm upgrade only reports the co degraded issue, didn't mention the invalid version issue.
# oc get clusterversion -oyaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
creationTimestamp: "2021-11-09T08:56:50Z"
generation: 5
name: version
resourceVersion: "516382"
uid: 51dd6fbb-966e-41f1-bd15-05cbde4cd5ad
spec:
channel: candidate-4.9
clusterID: 9331eba0-85a8-4a94-af81-739f89c70c97
desiredUpdate:
version: 4.9.6
status:
availableUpdates: null
conditions:
- lastTransitionTime: "2021-11-09T09:22:55Z"
message: Done applying 4.8.19
status: "True"
type: Available
- lastTransitionTime: "2021-11-10T07:46:16Z"
message: Cluster operator authentication is degraded
reason: ClusterOperatorDegraded
status: "True"
type: Failing
- lastTransitionTime: "2021-11-09T09:22:55Z"
message: 'Error while reconciling 4.8.19: the cluster operator authentication
is degraded'
reason: ClusterOperatorDegraded
status: "False"
type: Progressing
- lastTransitionTime: "2021-11-10T06:41:18Z"
message: 'Unable to retrieve available updates: currently reconciling cluster
version 4.8.19 not found in the "fast-4.9" channel'
reason: VersionNotFound
status: "False"
type: RetrievedUpdates
- lastTransitionTime: "2021-11-09T08:57:20Z"
message: |
Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see
the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.
reason: AdminAckRequired
status: "False"
type: Upgradeable
- lastTransitionTime: "2021-11-10T07:43:47Z"
message: 'The cluster version is invalid: spec.desiredUpdate.version: Invalid
value: "4.9.6": when image is empty the update must be a previous version
or an available update'
reason: InvalidClusterVersion
status: "True"
type: Invalid
desired:
image: quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59
url: https://access.redhat.com/errata/RHBA-2021:4109
version: 4.8.19
history:
- completionTime: "2021-11-09T09:22:55Z"
image: quay.io/openshift-release-dev/ocp-release@sha256:ac19c975be8b8a449dedcdd7520e970b1cc827e24042b8976bc0495da32c6b59
startedTime: "2021-11-09T08:56:50Z"
state: Completed
verified: false
version: 4.8.19
observedGeneration: 5
versionHash: oJVcBisP_Ao=
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Verifying it with:
# ./oc version
Client Version: 4.10.0-0.nightly-2021-11-09-181140
Server Version: 4.10.0-0.nightly-2021-11-09-181140
Kubernetes Version: v1.22.1+1b2affc
1. Install a 4.10 cluster
# ./oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.0-0.nightly-2021-11-09-181140 True False 6m20s Cluster version is 4.10.0-0.nightly-2021-11-09-181140
2. Make co degraded
cat <<EOF >oauth.yaml
apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
name: cluster
spec:
identityProviders:
- name: oidcidp
mappingMethod: claim
type: OpenID
openID:
clientID: test
clientSecret:
name: test
claims:
preferredUsername:
- preferred_username
name:
- name
email:
- email
issuer: https://www.idp-issuer.example.com
EOF
# ./oc apply -f oauth.yaml
oauth.config.openshift.io/cluster configured
# ./oc get co | grep auth
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.10.0-0.nightly-2021-11-09-181140 True False True 46m OAuthServerConfigObservationDegraded: failed to apply IDP oidcidp config: dial tcp: lookup www.idp-issuer.example.com on 172.30.0.10:53: no such host
Patch to an invalid desired version:
# ./oc patch clusterversion version --type json -p '[{"op": "add", "path": "/spec/channel", "value": "nightly-4.10"}, {"op": "add", "path": "/spec/desiredUpdate", "value": {"version": "4.10.1"}}]'
clusterversion.config.openshift.io/version patched
# ./oc get clusterversion/version -ojson | jq -r .status.conditions
[
{
"lastTransitionTime": "2021-11-12T02:27:44Z",
"message": "Done applying 4.10.0-0.nightly-2021-11-09-181140",
"status": "True",
"type": "Available"
},
{
"lastTransitionTime": "2021-11-12T03:16:59Z",
"message": "Cluster operator authentication is degraded",
"reason": "ClusterOperatorDegraded",
"status": "True",
"type": "Failing"
},
{
"lastTransitionTime": "2021-11-12T02:27:44Z",
"message": "Error while reconciling 4.10.0-0.nightly-2021-11-09-181140: the cluster operator authentication is degraded",
"reason": "ClusterOperatorDegraded",
"status": "False",
"type": "Progressing"
},
{
"lastTransitionTime": "2021-11-12T02:37:40Z",
"status": "True",
"type": "RetrievedUpdates"
},
{
"lastTransitionTime": "2021-11-12T03:21:35Z",
"message": "The cluster version is invalid: spec.desiredUpdate.version: Invalid value: \"4.10.1\": when image is empty the update must be a previous version or an available update",
"reason": "InvalidClusterVersion",
"status": "True",
"type": "Invalid"
}
]
We have Invalid version and co degraded conditions.
# ./oc adm upgrade
Error while reconciling 4.10.0-0.nightly-2021-11-09-181140: the cluster operator authentication is degraded
Upstream: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Channel: nightly-4.10
Available Updates:
VERSION IMAGE
4.10.0-0.nightly-2021-11-10-212548 registry.ci.openshift.org/ocp/release@sha256:b15acfa35c303c15148e1032774c91df0b38ea2b3efee4d8c408777d64467c70
4.10.0-0.nightly-2021-11-11-072405 registry.ci.openshift.org/ocp/release@sha256:4a4004a27b74f1f9a229755d9cb77701823ddbba4377bf090a1bfa4579e80d37
4.10.0-0.nightly-2021-11-11-170956 registry.ci.openshift.org/ocp/release@sha256:3b5eeefd3ba57ae2ffe81b34516ab7330fe966067a5ca467fb40d9476905b400
4.10.0-0.nightly-2021-11-12-023027 registry.ci.openshift.org/ocp/release@sha256:7635f6abdcff00ea285d8f85a4cacafed564bd9c9ecbe783cdc3afbc746f1b89
oc adm upgrade only prompts co degraded, doesn't mention invalid version issue.
Trevor, it seems oc still cannot prompt all the conditions. Could you please take a look? Thanks!
Attempted with Client Version: 4.10.0-0.nightly-2021-11-14-184249 and got the same results as comment#8. Re-opening it. checkForUpgrade (the function I adjusted in oc#812) is only called when --to, --to-image, or --to-latest is used. Simplest way to get there is probably to wait until I get the PR for bug 1992680 landed, and then set up the ClusterVersion object so it is Failing=True and Progressing=True at the same time. |