Bug 2006145
| Summary: | 4.8.12 to 4.9 upgrade hung due to cluster-version-operator pod CrashLoopBackOff: error creating clients: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> |
| Component: | Cluster Version Operator | Assignee: | W. Trevor King <wking> |
| Status: | CLOSED ERRATA | QA Contact: | Yang Yang <yanyang> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.9 | CC: | abhbaner, aos-bugs, jialiu, jokerman, wduan, wking, yanyang |
| Target Milestone: | --- | ||
| Target Release: | 4.9.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2005581 | Environment: | |
| Last Closed: | 2021-10-18 17:51:49 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2005581 | ||
| Bug Blocks: | |||
|
Description
W. Trevor King
2021-09-20 23:41:16 UTC
Verifying with 4.9.0-0.nightly-2021-09-21-215600. The CVO pod is rolled out to 4.9 successfully. There are 2 issues in this upgrade testing but they seems not relevant to this bug.
1. 2 nodes go to Notready
2. The old CVO pod is in terminating status but doesn't get removed
# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.12 True True 3h3m Working towards 4.9.0-0.nightly-2021-09-21-215600: 71 of 734 done (9% complete)
# oc get node
NAME STATUS ROLES AGE VERSION
compute-0 NotReady,SchedulingDisabled worker 4h13m v1.21.1+d8043e1
compute-1 Ready worker 4h13m v1.21.1+d8043e1
control-plane-0 Ready master 4h23m v1.21.1+d8043e1
control-plane-1 NotReady,SchedulingDisabled master 4h22m v1.21.1+d8043e1
control-plane-2 Ready master 4h23m v1.21.1+d8043e1
# oc get po -n openshift-cluster-version
NAME READY STATUS RESTARTS AGE
cluster-version-operator-df4858cf7-sm996 1/1 Running 0 122m
cluster-version-operator-df4858cf7-whbpx 1/1 Terminating 0 126m
version--rzg6k-k7x2v 0/1 Completed 0 3h4m
# oc get pod/cluster-version-operator-df4858cf7-sm996 -ojson | jq -r .spec.volumes[]
{
"hostPath": {
"path": "/etc/ssl/certs",
"type": ""
},
"name": "etc-ssl-certs"
}
{
"hostPath": {
"path": "/etc/cvo/updatepayloads",
"type": ""
},
"name": "etc-cvo-updatepayloads"
}
{
"name": "serving-cert",
"secret": {
"defaultMode": 420,
"secretName": "cluster-version-operator-serving-cert"
}
}
{
"name": "kube-api-access",
"projected": {
"defaultMode": 420,
"sources": [
{
"serviceAccountToken": {
"expirationSeconds": 3600,
"path": "token"
}
},
{
"configMap": {
"items": [
{
"key": "ca.crt",
"path": "ca.crt"
}
],
"name": "kube-root-ca.crt"
}
},
{
"downwardAPI": {
"items": [
{
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
},
"path": "namespace"
}
]
}
}
]
}
}
# oc get pod/cluster-version-operator-df4858cf7-qfg9n -ojson | jq -r .spec.containers[].volumeMounts
[
{
"mountPath": "/etc/ssl/certs",
"name": "etc-ssl-certs",
"readOnly": true
},
{
"mountPath": "/etc/cvo/updatepayloads",
"name": "etc-cvo-updatepayloads",
"readOnly": true
},
{
"mountPath": "/etc/tls/serving-cert",
"name": "serving-cert",
"readOnly": true
},
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access",
"readOnly": true
}
]
Moving it to verified state because the upgrade is not stuck on the CVO pod creation any more. *** Bug 2007230 has been marked as a duplicate of this bug. *** *** Bug 2007229 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |