Description of problem: The CI cluster for vSphere running in AWS stopped upgrading due to the network-check-target pod for a single worker that would not start. namespaces/openshift-kube-scheduler/pods/openshift-kube-scheduler-ip-10-0-55-206.us-west-2.compute.internal/kube-scheduler/kube-scheduler/logs/current.log:2021-02-23T16:30:00.602724318Z I0223 16:30:00.602688 1 factory.go:321] "Unable to schedule pod; no fit; waiting" pod="openshift-network-diagnostics/network-check-target-hn8nr" err="0/9 nodes are available: 1 Insufficient memory, 8 node(s) didn't match Pod's node affinity." $ oc get pod NAME READY STATUS RESTARTS AGE network-check-source-7b56ddbc7b-s5zxm 1/1 Running 0 98m ... network-check-target-hn8nr 0/1 Pending 0 98m workaround: 11077 2/23/2021 12:15 oc adm cordon ip-10-0-52-209.us-west-2.compute.internal 11078 2/23/2021 12:16 oc adm drain --force --ignore-daemonsets --delete-local-data ip-10-0-52-209.us-west-2.compute.internal 11079 2/23/2021 12:17 oc debug node/ip-10-0-52-209.us-west-2.compute.internal systemctl reboot Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The pod in question has a priority of 0, so it will not preempt other pods. This pod runs as part of a daemonset. Other system daemonsets run with a very high priority. For example, MachineConfigDaemon: priority: 2000001000 priorityClassName: system-node-critical If this component is required to run on each host, we should increase the priority. If this component is optional, we should determine a way to not block upgrades with the rollout of this component for capacity reasons.
Verified this bug on 4.8.0-0.nightly-2021-03-21-224928 $ oc get ds -n openshift-network-diagnostics -o yaml | grep crit networkoperator.openshift.io/non-critical: ""
hello, can this fix backport to OCP 4.7.X?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438