Description of problem: This happened during update from 4.5.14 to 4.6.0-rc.4. Nodes that were already part of the cluster were drained and rebooted to deploy the new machine configuration during the cluster upgrade. As a result, user workloads were rescheduled a number of times to other nodes in the cluster. Some of these workloads did not set CPU or RAM requests, and when they were all rescheduled to the same target node at once their thundering herd caused system OOMs. The kubelet on the node would become unresponsive after that time and remain in that state. Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Mon, 01 Jan 0001 00:00:00 +0000 Tue, 20 Oct 2020 11:16:12 -0700 RouteCreated openshift-sdn cleared kubelet-set NoRouteCreated MemoryPressure Unknown Wed, 21 Oct 2020 07:22:05 -0700 Wed, 21 Oct 2020 07:22:52 -0700 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Wed, 21 Oct 2020 07:22:05 -0700 Wed, 21 Oct 2020 07:22:52 -0700 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Wed, 21 Oct 2020 07:22:05 -0700 Wed, 21 Oct 2020 07:22:52 -0700 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Wed, 21 Oct 2020 07:22:05 -0700 Wed, 21 Oct 2020 07:22:52 -0700 NodeStatusUnknown Kubelet stopped posting node status. This is the last node that was stuck with NotReady. build0-gstfj-w-b-grtqb.c.openshift-ci-build-farm.internal NotReady worker 26d v1.19.0+d59ce34
This was diagnosed to an MCO issue.
No, this was never related to the MCO issue. This is an issue with the kubelet and/or the system responding to OOM events.
Applied the workaround: https://github.com/openshift/release/pull/13140 I will update here if it still happens. So far, it is good since last night.