Bug 1909600
| Summary: | Static pod installer controller deadlocks with non-existing installer pod, WAS: kube-apisrever of clsuter operator always with incorrect status due to pleg error | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Ke Wang <kewang> | |
| Component: | kube-apiserver | Assignee: | Stefan Schimanski <sttts> | |
| Status: | CLOSED ERRATA | QA Contact: | Ke Wang <kewang> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.7 | CC: | aos-bugs, cblecker, jokerman, jupierce, mfojtik, rgangwar, sanchezl, schoudha, sttts, travi, wking, xxia | |
| Target Milestone: | --- | |||
| Target Release: | 4.8.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | No Doc Update | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1858763 | |||
| : | 1949370 1995911 (view as bug list) | Environment: | ||
| Last Closed: | 2021-07-27 22:35:34 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1822018, 1858763 | |||
| Bug Blocks: | 1817419, 1822016, 1874597, 1949370 | |||
Tried the following workaround, cluster went back to normal.
oc patch etcd/cluster --type=json -p '[ {"op": "replace", "path": "/spec/forceRedeploymentReason", "value": "forced test 1" } ]'
oc patch kubeapiserver/cluster --type=json -p '[ {"op": "replace", "path": "/spec/forceRedeploymentReason", "value": "forced test 1" } ]'
I would need to see `kubectl get kubeapiservers -o yaml` before the workaround of comment 3. The one in the must-gather is on revision 97, not 8 where the issue happened. The installer pod is evicted by machine config daemon: quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-5d0410a767aeccfc527c7455bb828f07d56ae2849605f8796fcaba51ff838f08/namespaces/openshift-machine-config-operator/pods/machine-config-daemon-fq8j5/machine-config-daemon/machine-config-daemon/logs/current.log 47:2020-12-21T03:33:18.068185011Z E1221 03:33:18.068115 14063 daemon.go:342] WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-4pjpc, openshift-controller-manager/controller-manager-s7g27, openshift-dns/dns-default-zclmp, openshift-image-registry/node-ca-gnbgk, openshift-machine-config-operator/machine-config-daemon-fq8j5, openshift-machine-config-operator/machine-config-server-xxsmr, openshift-manila-csi-driver/csi-nodeplugin-nfsplugin-8pvpv, openshift-manila-csi-driver/openstack-manila-csi-nodeplugin-pq5rr, openshift-monitoring/node-exporter-mmntv, openshift-multus/multus-admission-controller-skdrr, openshift-multus/multus-x4vbt, openshift-multus/network-metrics-daemon-f992j, openshift-network-diagnostics/network-check-target-qdtgc, openshift-ovn-kubernetes/ovnkube-master-z6ns5, openshift-ovn-kubernetes/ovnkube-node-nfsgr, openshift-ovn-kubernetes/ovs-node-7wwqv; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 57:2020-12-21T03:33:18.077433978Z I1221 03:33:18.077397 14063 daemon.go:342] evicting pod openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 134:2020-12-21T03:33:37.157026872Z I1221 03:33:37.156888 14063 daemon.go:328] Evicted pod openshift-kube-apiserver/installer-8-juzhao-share-bbmw6-master-1 This looks like the cause of the missed installer. Checked back with MCO team about this eviction on reboot. This behaviour has not changed. So this BZ is not a blocker for 4.7 as far as we see. Nevertheless, work is ongoing to improve resilience to installer pod evictions (and other installer failures). Today, When we upgraded from 4.6.0-0.nightly-2021-03-21-131139--> 4.7.0-0.nightly-2021-03-22-025559, hit the same issue again.
cluster operator kube-apiserver error message:
...
Spec:
Status:
Conditions:
Last Transition Time: 2021-03-24T03:53:37Z
Message: NodeInstallerDegraded: 1 nodes are failing on revision 11:
NodeInstallerDegraded: no detailed termination message, see `oc get -n "openshift-kube-apiserver" pods/"installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal" -oyaml`
Reason: NodeInstaller_InstallerPodFailed
Status: True
Type: Degraded
Last Transition Time: 2021-03-24T03:51:13Z
Message: NodeInstallerProgressing: 3 nodes are at revision 10; 0 nodes have achieved new revision 11
Reason: NodeInstaller
Status: True
Type: Progressing
Last Transition Time: 2021-03-24T01:39:36Z
Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 10; 0 nodes have achieved new revision 11
Reason: AsExpected
Status: True
Type: Available
Last Transition Time: 2021-03-24T01:35:51Z
Message: All is well
Reason: AsExpected
Status: True
Type: Upgradeable
must-gather.local.5989955761737491992/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-de5a26ae6ddea0ad63182c508ca5e730bdc14c52daffa5c42d9a718aa7720e9b/namespaces/openshift-machine-config-operator/pods
# grep -n 'installer-11-yinzhou240913' machine-config-daemon-sldps/machine-config-daemon/machine-config-daemon/logs/current.log
37:2021-03-24T03:51:28.901402277Z E0324 03:51:28.901336 103262 daemon.go:344] WARNING: ignoring DaemonSet-managed Pods: openshift-cluster-csi-drivers/gcp-pd-csi-driver-node-f6pp4, openshift-cluster-node-tuning-operator/tuned-64sx8, openshift-controller-manager/controller-manager-77n7g, openshift-dns/dns-default-tptbh, openshift-image-registry/node-ca-zf5mz, openshift-machine-config-operator/machine-config-daemon-sldps, openshift-machine-config-operator/machine-config-server-4jk7s, openshift-monitoring/node-exporter-kj9l8, openshift-multus/multus-admission-controller-wm6gb, openshift-multus/multus-lnbwb, openshift-multus/network-metrics-daemon-rthb2, openshift-network-diagnostics/network-check-target-gqblf, openshift-ovn-kubernetes/ovnkube-master-rfpvh, openshift-ovn-kubernetes/ovnkube-node-6h5c7, openshift-ovn-kubernetes/ovs-node-qp6vg; deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal
46:2021-03-24T03:51:28.941813425Z I0324 03:51:28.941788 103262 daemon.go:344] evicting pod openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal
91:2021-03-24T03:51:46.256188435Z I0324 03:51:46.254499 103262 daemon.go:330] Evicted pod openshift-kube-apiserver/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal
106:2021-03-24T03:53:10.714342933Z I0324 03:53:10.714234 103262 daemon.go:344] evicting pod openshift-kube-controller-manager/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal
108:2021-03-24T03:53:10.775250978Z I0324 03:53:10.775195 103262 daemon.go:330] Evicted pod openshift-kube-controller-manager/installer-11-yinzhou240913-vfdzl-m-1.c.openshift-qe.internal
In kubelet log, found PLEG error,
$ grep 'PLEG is not healthy' *
kubelet_service.log:Mar 24 02:49:53.776520 yinzhou240913-vfdzl-m-0.c.openshift-qe.internal hyperkube[1725]: E0324 02:49:53.776332 1725 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
kubelet_service.log:Mar 24 02:55:13.930773 yinzhou240913-vfdzl-m-1.c.openshift-qe.internal hyperkube[1712]: E0324 02:55:13.930693 1712 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
kubelet_service.log:Mar 24 02:59:43.970704 yinzhou240913-vfdzl-m-2.c.openshift-qe.internal hyperkube[1748]: E0324 02:59:43.969052 1748 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
kubelet_service.log:Mar 24 03:54:43.225651 yinzhou240913-vfdzl-m-1.c.openshift-qe.internal hyperkube[1747]: E0324 03:54:43.225600 1747 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
kubelet_service.log:Mar 24 04:00:55.054332 yinzhou240913-vfdzl-m-0.c.openshift-qe.internal hyperkube[1734]: E0324 04:00:55.054119 1734 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
kubelet_service.log:Mar 24 04:07:24.362134 yinzhou240913-vfdzl-m-2.c.openshift-qe.internal hyperkube[1756]: E0324 04:07:24.362105 1756 kubelet.go:1806] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Ke Wang added UpgradeBlocker on the 24th, which sucks this bug into the update-blocker lifecycle [1]. So we're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the ImpactStatementRequested label has been added to this bug. When responding, please remove ImpactStatementRequested and set the ImpactStatementProposed label. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? * example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet * example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? * example: Up to 2 minute disruption in edge routing * example: Up to 90 seconds of API downtime * example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? * example: Issue resolves itself after five minutes * example: Admin uses oc to fix things * example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? * example: No, it has always been like this we just never noticed * example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1 [1]: https://github.com/openshift/enhancements/pull/475 > Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? We have one upgrade profile Disconnected IPI on vSphere 7.0 with RHCOS & http_proxy from 4.6.z to 4.7.z, 100% hit this bug. > What is the impact? Is it serious enough to warrant blocking edges? Only impact vSphere. > How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? Not sure if it is serious impact, no one solution to mitigate now, will plan to have new upgrade test with latest 4.7.z payload with the same upgrade profile. > Is this a regression? Yes, it is cloned form bug 1858763 on 4.6. No longer blocker issue. Upgraded OCP vSphere disconnected cluster from 4.6.24 to 4.7.nightly，then to 4.8 nightly successfully.
$ oc get infrastructures.config.openshift.io -o yaml | grep -A1 " platformSpec"
platformSpec:
type: VSphere
$ oc get clusterversion -o json|jq ".items[0].status.history"
[
{
"completionTime": "2021-04-13T10:32:35Z",
"image": "registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-04-09-222447",
"startedTime": "2021-04-13T09:15:24Z",
"state": "Completed",
"verified": false,
"version": "4.8.0-0.nightly-2021-04-09-222447"
},
{
"completionTime": "2021-04-13T08:29:09Z",
"image": "registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-04-10-082109",
"startedTime": "2021-04-13T07:02:37Z",
"state": "Completed",
"verified": false,
"version": "4.7.0-0.nightly-2021-04-10-082109"
},
{
"completionTime": "2021-04-13T07:01:52Z",
"image": "registry.ci.openshift.org/ocp/release@sha256:d422c5f6a531374dc03484bb0adaeb26b7b179dec055e15ecd6042cc750dd140",
"startedTime": "2021-04-13T07:01:07Z",
"state": "Completed",
"verified": false,
"version": "4.6.24"
},
{
"completionTime": "2021-04-13T07:01:07Z",
"image": "registry.ci.openshift.org/ocp/release@sha256:a78b87ed93b759ceed7f62bc6c477ba6a3927d34d82a18f56fe2fa3147b2fd3e",
"startedTime": "2021-04-13T06:54:22Z",
"state": "Partial",
"verified": false,
"version": "4.7.0-0.nightly-2021-04-10-082109"
},
{
"completionTime": "2021-04-13T04:45:52Z",
"image": "registry.ci.openshift.org/ocp/release@sha256:d422c5f6a531374dc03484bb0adaeb26b7b179dec055e15ecd6042cc750dd140",
"startedTime": "2021-04-13T03:42:34Z",
"state": "Completed",
"verified": true,
"version": "4.6.24"
},
{
"completionTime": "2021-04-13T03:12:29Z",
"image": "quay.io/openshift-release-dev/ocp-release@sha256:5c8ab6c4a863f9ac077bc743579d9387bb7cd311a36c3197e609f8be63d17981",
"startedTime": "2021-04-13T02:44:51Z",
"state": "Completed",
"verified": false,
"version": "4.6.23"
}
]
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.nightly-2021-04-09-222447 True False 21m Cluster version is 4.8.0-0.nightly-2021-04-09-222447
$ oc get no
NAME STATUS ROLES AGE VERSION
kewang-134601-lnvpv-master-0 Ready master 8h v1.20.0+5f82cdb
kewang-134601-lnvpv-master-1 Ready master 8h v1.20.0+5f82cdb
kewang-134601-lnvpv-master-2 Ready master 8h v1.20.0+5f82cdb
kewang-134601-lnvpv-worker-25n5k Ready worker 7h56m v1.20.0+5f82cdb
kewang-134601-lnvpv-worker-bdjnz Ready worker 7h56m v1.20.0+5f82cdb
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.8.0-0.nightly-2021-04-09-222447 True False False 27m
baremetal 4.8.0-0.nightly-2021-04-09-222447 True False False 3h26m
cloud-credential 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
cluster-autoscaler 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
config-operator 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
console 4.8.0-0.nightly-2021-04-09-222447 True False False 34m
csi-snapshot-controller 4.8.0-0.nightly-2021-04-09-222447 True False False 34m
dns 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
etcd 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
image-registry 4.8.0-0.nightly-2021-04-09-222447 True False False 44m
ingress 4.8.0-0.nightly-2021-04-09-222447 True False False 7h59m
insights 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
kube-apiserver 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
kube-controller-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
kube-scheduler 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
kube-storage-version-migrator 4.8.0-0.nightly-2021-04-09-222447 True False False 44m
machine-api 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
machine-approver 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
machine-config 4.8.0-0.nightly-2021-04-09-222447 True False False 26m
marketplace 4.8.0-0.nightly-2021-04-09-222447 True False False 41m
monitoring 4.8.0-0.nightly-2021-04-09-222447 True False False 32m
network 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
node-tuning 4.8.0-0.nightly-2021-04-09-222447 True False False 72m
openshift-apiserver 4.8.0-0.nightly-2021-04-09-222447 True False False 27m
openshift-controller-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 3h22m
openshift-samples 4.8.0-0.nightly-2021-04-09-222447 True False False 72m
operator-lifecycle-manager 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
operator-lifecycle-manager-catalog 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
operator-lifecycle-manager-packageserver 4.8.0-0.nightly-2021-04-09-222447 True False False 34m
service-ca 4.8.0-0.nightly-2021-04-09-222447 True False False 8h
storage 4.8.0-0.nightly-2021-04-09-222447 True False False 34m
All is fine, so move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 Based on the impact statement in comment 12 we have not blocked update recommendations based on this bug. If folks bump into it more frequently, feel free to restore the UpgradeBlocker keyword (which was already removed after comment 13). |
This problem was fixed on 4.6, bug reproduced on OCP 4.7 4.7.0-0.nightly-2020-12-20-031835, $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-12-20-031835 True False 160m Error while reconciling 4.7.0-0.nightly-2020-12-20-031835: the cluster operator kube-apiserver is degraded $ oc get no NAME STATUS ROLES AGE VERSION juzhao-share-bbmw6-master-0 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-master-1 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-master-2 Ready master 3h13m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-bhprx Ready worker 179m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-fs2h5 Ready worker 175m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-p48wh Ready worker 178m v1.20.0+87544c5 juzhao-share-bbmw6-worker-0-qxktd Ready worker 176m v1.20.0+87544c5 $ oc get co/kube-apiserver NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE kube-apiserver 4.7.0-0.nightly-2020-12-20-031835 True True True 3h16m $ oc describe co/kube-apiserver Name: kube-apiserver Namespace: ... Spec: Status: Conditions: Last Transition Time: 2020-12-21T03:35:21Z Message: NodeInstallerDegraded: 1 nodes are failing on revision 8: NodeInstallerDegraded: no detailed termination message, see `oc get -n "openshift-kube-apiserver" pods/"installer-8-juzhao-share-bbmw6-master-1" -oyaml` Reason: NodeInstaller_InstallerPodFailed Status: True Type: Degraded Last Transition Time: 2020-12-21T03:32:06Z Message: NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8 Reason: NodeInstaller Status: True Type: Progressing Last Transition Time: 2020-12-21T03:02:45Z Message: StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8 Reason: AsExpected Status: True Type: Available Last Transition Time: 2020-12-21T02:59:28Z Message: All is well Reason: AsExpected Status: True Type: Upgradeable Extension: <nil> ... $ oc get pods -n openshift-kube-apiserver -l apiserver --show-labels NAME READY STATUS RESTARTS AGE LABELS kube-apiserver-juzhao-share-bbmw6-master-0 5/5 Running 0 3h17m apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-juzhao-share-bbmw6-master-1 5/5 Running 0 3h4m apiserver=true,app=openshift-kube-apiserver,revision=6 kube-apiserver-juzhao-share-bbmw6-master-2 5/5 Running 0 3h22m apiserver=true,app=openshift-kube-apiserver,revision=6 $ oc logs -n openshift-kube-apiserver-operator kube-apiserver-operator-5c77c9c859-k684x | grep -n '1 nodes are failing on revision 8' 74:I1221 03:36:04.463569 1 status_controller.go:213] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2020-12-21T03:35:21Z","message":"NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled","reason":"NodeInstaller_InstallerPodFailed::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-12-21T03:32:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-12-21T03:02:45Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-12-21T02:59:28Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} 85:I1221 03:36:04.614567 1 status_controller.go:213] clusteroperator/kube-apiserver diff {"status":{"conditions":[{"lastTransitionTime":"2020-12-21T03:35:21Z","message":"NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled","reason":"NodeInstaller_InstallerPodFailed::StaticPods_Error","status":"True","type":"Degraded"},{"lastTransitionTime":"2020-12-21T03:32:06Z","message":"NodeInstallerProgressing: 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"NodeInstaller","status":"True","type":"Progressing"},{"lastTransitionTime":"2020-12-21T03:02:45Z","message":"StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 6; 0 nodes have achieved new revision 8","reason":"AsExpected","status":"True","type":"Available"},{"lastTransitionTime":"2020-12-21T02:59:28Z","message":"All is well","reason":"AsExpected","status":"True","type":"Upgradeable"}]}} 86:I1221 03:36:04.621298 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"d6219038-bd2e-4744-98ef-9cf9a42858e7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`" to "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled" 107:I1221 03:36:04.807779 1 event.go:282] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-kube-apiserver-operator", Name:"kube-apiserver-operator", UID:"d6219038-bd2e-4744-98ef-9cf9a42858e7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/kube-apiserver changed: Degraded message changed from "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`" to "NodeInstallerDegraded: 1 nodes are failing on revision 8:\nNodeInstallerDegraded: no detailed termination message, see `oc get -n \"openshift-kube-apiserver\" pods/\"installer-8-juzhao-share-bbmw6-master-1\" -oyaml`\nStaticPodsDegraded: context canceled" ...