Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: Worker node is in Ready,SchedulingDisabled state Version-Release number of selected component (if applicable): HCO image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-hyperconverged-cluster-operator@sha256:6ec162f13eb3b152168a62e31a4d0fc9f36a481901fe430fab4ebdfcd50ff7a7 CSV creation time: 2020-06-03 02:37:06 OpenShift: 4.5.0-0.nightly-2020-06-02-220939 How reproducible: always Steps to Reproduce: 1. deploy cnv 2. 3. Actual results: Worker node is in Ready,SchedulingDisabled state oc logs kubemacpool-mac-controller-manager-7576759bff-pfhxr -nopenshift-cnv {"level":"info","ts":1591140668.2270849,"logger":"manager","msg":"Setting up client for manager"} {"level":"info","ts":1591140668.2320285,"logger":"manager","msg":"Setting up leader electionManager"} {"level":"info","ts":1591140675.0391695,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1591140675.048264,"logger":"manager","msg":"waiting for manager to become leader"} I0602 23:31:15.048519 1 leaderelection.go:241] attempting to acquire leader lease openshift-cnv/kubemacpool-election... oc describe node kwc-nightly-tnfmp-worker-72bk9 machineconfiguration.openshift.io/desiredConfig: rendered-worker-ff7a754ee346afe6375caeadc85b5cab machineconfiguration.openshift.io/reason: failed to drain node (5 tries): timed out waiting for the condition: error when evicting pod "kubemacpool-mac-controller-manager-7576759bf... machineconfiguration.openshift.io/state: Degraded oc describe pod kubemacpool-mac-controller-manager-7576759bff-pfhxr -nopenshift-cnv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 4m1s (x4168 over 11h) kubelet, kwc-nightly-tnfmp-worker-6b6cl Readiness probe failed: Get https://10.131.0.12:8000/readyz: dial tcp 10.131.0.12:8000: connect: connection refused Expected results: Additional info:
The problem is that `oc adm drain myakove-j6fhs-worker-kdpbt --delete-local-data --ignore-daemonsets=true --force` fails. kubemacpool works in active-backup approach so pod disruption budget of min 1 pod should be removed. fixed in https://github.com/k8snetworkplumbingwg/kubemacpool/pull/169
Test Environment : ================== $ oc version Client Version: 4.5.0-rc.1 Server Version: 4.5.0-rc.1 Kubernetes Version: v1.18.3+a637491 CNV Version $ oc get csv -n openshift-cnv | awk ' { print $4 } ' | tail -n1 2.4.0 Steps: ===== Bug Summary: draining nodes with kubemacpool-mac-controller-manager pods on them failed and couldn't evict those pods. also, the node couldn't be rescheduled and was stuck at Ready,SchedulingDisabled mode. trying to uncordon it would look ok but after a little while was back to Ready,SchedulingDisabled. Fix: nodes with kubemacpool-mac-controller-manager on them will drain properly and will be able to reschedule. 1) see that the nodes are Ready $ oc get nodes NAME STATUS ROLES AGE VERSION host-172-16-0-15 Ready master 38h v1.18.3+a637491 host-172-16-0-19 Ready master 38h v1.18.3+a637491 host-172-16-0-25 Ready master 38h v1.18.3+a637491 host-172-16-0-27 Ready worker 38h v1.18.3+a637491 host-172-16-0-34 Ready worker 38h v1.18.3+a637491 2) check which nodes has kubemacpool-mac-controller-manager pods on them $ oc get pod -n openshift-cnv -o wide | grep kubemacpool-mac-controller-manager kubemacpool-mac-controller-manager-57b6cc7986-9zzdh 0/1 Running 0 7m12s 10.128.2.199 host-172-16-0-34 <none> <none> kubemacpool-mac-controller-manager-57b6cc7986-jpbnt 1/1 Running 0 11m 10.128.2.188 host-172-16-0-34 <none> <none> 3) drain the node and see that the kubemacpool-mac-controller-manager pods were evicted and that the drain succeeded $ oc adm drain host-172-16-0-34 --delete-local-data --ignore-daemonsets=true --force .. pod/kubemacpool-mac-controller-manager-57b6cc7986-jpbnt evicted .. pod/kubemacpool-mac-controller-manager-57b6cc7986-9zzdh evicted .. node/host-172-16-0-34 evicted 4) see the node is SchedulingDisabled and uncordon it $ oc get nodes NAME STATUS ROLES AGE VERSION host-172-16-0-15 Ready master 39h v1.18.3+a637491 host-172-16-0-19 Ready master 39h v1.18.3+a637491 host-172-16-0-25 Ready master 39h v1.18.3+a637491 host-172-16-0-27 Ready worker 38h v1.18.3+a637491 host-172-16-0-34 Ready,SchedulingDisabled worker 38h v1.18.3+a637491 $ oc adm uncordon host-172-16-0-34 node/host-172-16-0-34 uncordoned 5) wait about a minute or two and check that it is in Ready mode $ oc get nodes NAME STATUS ROLES AGE VERSION host-172-16-0-15 Ready master 39h v1.18.3+a637491 host-172-16-0-19 Ready master 39h v1.18.3+a637491 host-172-16-0-25 Ready master 39h v1.18.3+a637491 host-172-16-0-27 Ready worker 39h v1.18.3+a637491 host-172-16-0-34 Ready worker 39h v1.18.3+a637491
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194