Description of problem: Applying node taints (wit NoEcecute) to a compute node in tha clustr not only prevents pods with toleration of the taint to be scheduled, but actuall kills critcal pods like k8s_sync_sync, k8s_sdn_sdn, k8s_openvswitch_ovs Version-Release number of selected component (if applicable): # oc version oc v3.10.0+0c4577e-1 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://gid-okd-blue-master.uio.no:8443 openshift v3.10.0+456651d-57 kubernetes v1.10.0+b81c8f8 How reproducible: Steps to Reproduce: Cluster nodes: [root@sm-linux uio-kibana]# oc get nodes NAME STATUS ROLES AGE VERSION gid-okd-blue-m.uio.no Ready master 6d v1.10.0+b81c8f8 gid-okd-blue01.uio.no Ready infra 6d v1.10.0+b81c8f8 gid-okd-blue02.uio.no Ready compute 6d v1.10.0+b81c8f8 gid-okd-blue03.uio.no Ready compute 6d v1.10.0+b81c8f8 [root@sm-linux uio-kibana]# 1. Check that critical pods are running. [root@gid-okd-blue03 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c0c766a83c81 cf75e4641dbb "/bin/bash -c '#!/..." 20 minutes ago Up 20 minutes k8s_sync_sync-qml2m_openshift-node_ed5e4991-e8e3-11e8-89a7-005056a958a7_0 94c35ad94714 cf75e4641dbb "/bin/bash -c '#!/..." 20 minutes ago Up 20 minutes k8s_sdn_sdn-tfgxh_openshift-sdn_ed590808-e8e3-11e8-89a7-005056a958a7_0 a12235d0a45f cf75e4641dbb "/bin/bash -c '#!/..." 20 minutes ago Up 20 minutes k8s_openvswitch_ovs-nw29t_openshift-sdn_ed57f39a-e8e3-11e8-89a7-005056a958a7_0 bf9595b735f8 harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 20 minutes ago Up 20 minutes k8s_POD_sync-qml2m_openshift-node_ed5e4991-e8e3-11e8-89a7-005056a958a7_0 4ab268bbb0c0 harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 20 minutes ago Up 20 minutes k8s_POD_sdn-tfgxh_openshift-sdn_ed590808-e8e3-11e8-89a7-005056a958a7_0 6796e9141be7 harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 20 minutes ago Up 20 minutes k8s_POD_ovs-nw29t_openshift-sdn_ed57f39a-e8e3-11e8-89a7-005056a958a7_0 [root@gid-okd-blue03 ~]# 2. taint one of the compute nodes. [root@sm-linux uio-kibana]# oc adm taint nodes gid-okd-blue03.uio.no esAccess=true:NoExecute node "gid-okd-blue03.uio.no" tainted [root@sm-linux uio-kibana]# 3.Check running pods/containers again [root@gid-okd-blue03 ~]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@gid-okd-blue03 ~]# 4. Untaint the node [root@sm-linux uio-kibana]# oc adm taint nodes gid-okd-blue03.uio.no esAccess- node "gid-okd-blue03.uio.no" untainted [root@sm-linux uio-kibana]# 5. Chack that critical pods are up and running again. [root@gid-okd-blue03 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 35e9670d42b6 cf75e4641dbb "/bin/bash -c '#!/..." 54 seconds ago Up 53 seconds k8s_sdn_sdn-8cpqg_openshift-sdn_59742401-e8e7-11e8-89a7-005056a958a7_0 deddf455e52e cf75e4641dbb "/bin/bash -c '#!/..." 54 seconds ago Up 53 seconds k8s_openvswitch_ovs-t6s7v_openshift-sdn_596df7de-e8e7-11e8-89a7-005056a958a7_0 9f923cdab1c0 cf75e4641dbb "/bin/bash -c '#!/..." 54 seconds ago Up 53 seconds k8s_sync_sync-nv5dm_openshift-node_596b7faf-e8e7-11e8-89a7-005056a958a7_0 24443c9d6b36 harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 54 seconds ago Up 54 seconds k8s_POD_sdn-8cpqg_openshift-sdn_59742401-e8e7-11e8-89a7-005056a958a7_0 91131b93d47c harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 54 seconds ago Up 54 seconds k8s_POD_ovs-t6s7v_openshift-sdn_596df7de-e8e7-11e8-89a7-005056a958a7_0 e39fb79bd42f harbor.uio.no/okd/origin-pod:v3.10.0 "/usr/bin/pod" 54 seconds ago Up 54 seconds k8s_POD_sync-nv5dm_openshift-node_596b7faf-e8e7-11e8-89a7-005056a958a7_0 [root@gid-okd-blue03 ~]# Actual results: Expected results: Additional info:
Hi Jarle, The pods you mentioned even though critical have no toleration for `NoExecute` taint. To be clear, in general critical pods doesn't have tolerations unless explicitly specified in the pod spec or created through a DS. Are you noticing any issue with those pods not having the toleration? I am curious about your use-case here, is this causing an upgrade or some other cluster functionality to fail?
Hi Ravig, I expected that critical pods, like the pods running core cluster node services (sdn,ovs, and so on) would have all tolerations provided by the installation. I.e. i did not expect the node to be offline when adding those taints. It does not help to add tolerations to pods from user-deployments. It won't schedule anythning on the node because the node is no longer running the required services to be able to receive _any_ pods.
The ovs and sdn issues appear to be fixed by OVS: tolerate taints #10731 https://github.com/openshift/openshift-ansible/pull/10731/files openshift-merge-robot merged 1 commit into openshift:release-3.11 from vrutkovs:tolerate-ovs on Nov 20, 2018 SDN: tolerate taints #11550 https://github.com/openshift/openshift-ansible/pull/11550 openshift-merge-robot merged 1 commit into openshift:release-3.11 from vrutkovs:3.11-sdn-tolerations on Apr 24 commit aea7524b8b20b59b0238feb9df36a3b2de413dab (HEAD, upstream/pr/10735) Author: Vadim Rutkovsky <roignac> Date: Tue Nov 20 15:55:46 2018 +0100 OVS: tolerate taints Could you try a recent version?
$ oc version ... Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-06-14T18:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} $ oc adm taint node node.lab.variantweb.net dedicated=special-user:NoExecute node/node.lab.variantweb.net tainted $ oc get node node.lab.variantweb.net -ojson | jq '.spec.taints' [ { "effect": "NoExecute", "key": "dedicated", "value": "special-user" } ] $ ogpa -o wide | grep node.lab openshift-node sync-w54sm 1/1 Running 0 18m 10.42.10.208 node.lab.variantweb.net <none> openshift-sdn ovs-vlfb5 1/1 Running 0 18m 10.42.10.208 node.lab.variantweb.net <none> openshift-sdn sdn-l9g5h 1/1 Running 0 18m 10.42.10.208 node.lab.variantweb.net <none> I do see that the node-exporter is still evicted though. That's an issue for monitoring though. This bug is fixed.
fyi opened https://bugzilla.redhat.com/show_bug.cgi?id=1720758 to track node-exporter bug
Verified this bug on v3.10.153 all openshift-sdn pod are working even if esAccess=true:NoExecute
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1753