This bug was initially created as a copy of Bug #1728873 I am copying this bug because: I've had a node which has been in 'SchedulingDisabled' state for a week due to the following annotations on the node: ``` Annotations: machine.openshift.io/machine: openshift-machine-api/02cd6e91-gn7q6-worker-us-west-1b-j8rn8 machineconfiguration.openshift.io/currentConfig: rendered-worker-7f899c2c109ce7eea3cde169e3f51092 machineconfiguration.openshift.io/desiredConfig: rendered-worker-71095e466f5b906cdbcc9ff8cb04414c machineconfiguration.openshift.io/reason: during bootstrap: pending config rendered-worker-71095e466f5b906cdbcc9ff8cb04414c bootID e4144af2-e895-44b6-97f1-2b205684e956 matches curr... ``` Manually rebooting the host fixed the issue (because the boot id no longer matched and it re-ran the reconciliation loop on the currentConfig)
OK Micah (or anyone else QE who wants to verify this fix), the reproducer here is to start rolling out any change via the MCO (could be an `oc adm upgrade`, or simpler, just create a /etc/dummy file via a MachineConfig), and use `oc logs pod/machine-config-daemon-xyz` on a MCD to watch when it starts to drain. Next, use `oc rsh pod/machine-config-daemon-xyz` and e.g. `pkill -f machine-config-daemon` or so. On clusters without this bug fixed, you should see the MCD stuck with the error mentioned in the BZ. If the bug is fixed, the MCD will log a message and re-start the drain.
Micah, can you help verify this to allow the 4.1 & 4.1 PR to get in?
I can still get the node stuck in scheduling disabled state. Reproducer steps: 1. create 50 pods in a node to slow down the drain for i in {1..50}; do oc run --restart=Never --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "YOUR_HOST_NAME" } } }' --image registry.fedoraproject.org/fedora:30 "zzz$i" -- sleep 1h; sleep .001; done 2. in another terminal watch -n 1 oc get node/YOUR_NODE_NAME -o yaml 3. in original terminal -- add a file with mco cat file.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: test-file spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=utf;base64,c2VydmVyIGZvby5leGFtcGxlLm5ldCBtYXhkZWxheSAwLjQgb2ZmbGluZQpzZXJ2ZXIgYmFyLmV4YW1wbGUubmV0IG1heGRlbGF5IDAuNCBvZmZsaW5lCnNlcnZlciBiYXouZXhhbXBsZS5uZXQgbWF4ZGVsYXkgMC40IG9mZmxpbmUK filesystem: root mode: 0644 path: /etc/test oc apply -f file.yaml 4. quickly oc debug node/YOUR_NODE_NAME, chroot /host, pkill -9 -f machine-config-daemon 5. check the watch output in step 2 to see that it gets stuck in SchedulingDisabled Note: If you don't move quickly enough the nodes will drain, the file will be added, and the system will reboot before you can pkill mcd. If that is the case then create more pods in step 1 or repeat the steps faster (remember to change the file added). Running: 4.2.0-0.nightly-2019-07-30-073644
> 4. quickly oc debug node/YOUR_NODE_NAME, chroot /host, pkill -9 -f machine-config-daemon are we super sure this is actually killing the MCD? my very latest tests ~1 week ago didn't result in MCD being killed but I may have made some mistakes
(In reply to Antonio Murdaca from comment #5) > > 4. quickly oc debug node/YOUR_NODE_NAME, chroot /host, pkill -9 -f machine-config-daemon > > are we super sure this is actually killing the MCD? my very latest tests ~1 > week ago didn't result in MCD being killed but I may have made some mistakes sh-4.4# ps -aux | grep /usr/bin/machine-config-daemon root 1542 0.0 0.4 135068 33012 ? Ssl 16:20 0:00 /usr/bin/machine-config-daemon start root 49363 0.0 0.0 9184 1044 ? S+ 16:38 0:00 grep /usr/bin/machine-config-daemon sh-4.4# sh-4.4# sh-4.4# until journalctl -b | grep 'beginning drain'; do sleep .01 ; done; date; pkill -9 -f machine-conig-daemon && echo 'killed mcd'; watch -n 1 'ps -ef | grep /usr/bin/machine-config-daemon' Aug 07 16:40:21 ip-10-0-144-145 root[54394]: machine-config-daemon[1542]: Update prepared; beginning drain Wed Aug 7 16:40:21 UTC 2019 killed mcd sh-4.4# ps -ef | grep /usr/bin/machine-config-daemon root 56707 46030 0 16:41 ? 00:00:00 grep /usr/bin/machine-config-daemon sh-4.4# date Wed Aug 7 16:41:37 UTC 2019 sh-4.4# ps -ef | grep /usr/bin/machine-config-daemon; date root 57688 46030 0 16:41 ? 00:00:00 grep /usr/bin/machine-config-daemon Wed Aug 7 16:41:42 UTC 2019 sh-4.4# ps -ef | grep /usr/bin/machine-config-daemon; date root 65770 46030 0 16:45 ? 00:00:00 grep /usr/bin/machine-config-daemon Wed Aug 7 16:45:12 UTC 2019 sh-4.4#
Ok I managed to finally reproduce and catch the log line that we need in order to verify this bug. I've used Michael's instructions + an additional kubectl delete, here's how: 1. in a tab: "watch oc get nodes" 2. in another tab: "oc edit mcp/worker" -> add "maxUnavailable: 3" to the spec, save and exit 3. create 200 pods in a worker node to slow down the drain and note down the node name for i in {1..200}; do oc run --restart=Never --overrides='{ "spec": { "nodeSelector": { "kubernetes.io/hostname": "NODE_NAME" } } }' --image registry.fedoraproject.org/fedora:30 "zzz$i" -- sleep 1h; sleep .001; done 4. identify the MCD running on NODE_NAME: oc get pods -l k8s-app=machine-config-daemon --field-selector spec.nodeName=NODE_NAME 5. in another tab: "oc logs -f MCD_FROM_NODE_NAME" (this will tell when the drain begins) 6. jump on NODE_NAME in yet another tab, then chroot /host oc debug node/NODE_NAME 7. now finally create a dummy MC: cat file.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: test-file spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=utf;base64,c2VydmVyIGZvby5leGFtcGxlLm5ldCBtYXhkZWxheSAwLjQgb2ZmbGluZQpzZXJ2ZXIgYmFyLmV4YW1wbGUubmV0IG1heGRlbGF5IDAuNCBvZmZsaW5lCnNlcnZlciBiYXouZXhhbXBsZS5uZXQgbWF4ZGVsYXkgMC40IG9mZmxpbmUK filesystem: root mode: 0644 path: /etc/test oc create -f file.yaml 8. as soon as the MC rolls out, you can see nodes going SchedulingDisabled, you should also see the MCD logs from point [5] flowing till drain starts 9. as soon as drain start on NODE_NAME from the MCD logs, jump in the shell from point [6] and kill the MCD: pkill -9 -f machine-config-daemon 10. verify no mcd anymore: ps -aux | grep /usr/bin/machine-config-daemon root 41141 0.0 0.0 9184 1080 ? S+ 11:17 0:00 grep /usr/bin/machine-config-daemon 11. kill the pod instance in kubernetes: oc delete pod -n openshift-machine-config-operator MCD_FOR_NODE_NAME 12. wait for a new instance with a different name to be re-spawned 13. when you have the new MCD instance, `oc logs` on it 14. Verify "drain interrupted, retrying" log is there 15. from the shell in [6], also verify the log is in the journal: sh-4.4# journalctl -b | grep "drain interrupted" Aug 28 11:19:13 ip-10-0-170-170 root[43929]: machine-config-daemon[43176]: drain interrupted, retrying 16. all good, now just watch the worker pool reconcile itself (it could take some time of course...so be patient) Let me know if you need further assistance as well!
It worked! Verified on 4.2.0-0.nightly-2019-08-28-083236 -- snip -- I0828 20:20:45.281186 3053 update.go:984] Update prepared; beginning drain I0828 20:20:45.290971 3053 update.go:89] cordoned node "ip-10-0-133-89.us-west-2.compute.internal" I0828 20:20:49.861927 3053 update.go:93] deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: zzz1, zzz10, zzz100, zzz101, zzz102, zzz103, zzz104, zzz105, zzz106, zzz107, zzz108, zzz109, zzz11, zzz110, zzz111, zzz112, zzz113, zzz114, zzz115, zzz116, zzz117, zzz118, zzz119, zzz12, zzz120, zzz121, zzz122, zzz123, zzz124, zzz125, zzz126, zzz127, zzz128, zzz129, zzz13, zzz130, zzz131, zzz132, zzz133, zzz134, zzz135, zzz136, zzz137, zzz138, zzz139, zzz14, zzz140, zzz141, zzz142, zzz143, zzz144, zzz145, zzz146, zzz147, zzz148, zzz149, zzz15, zzz150, zzz151, zzz152, zzz153, zzz154, zzz155, zzz156, zzz157, zzz158, zzz159, zzz16, zzz160, zzz161, zzz162, zzz163, zzz164, zzz165, zzz166, zzz167, zzz168, zzz169, zzz17, zzz170, zzz171, zzz172, zzz173, zzz174, zzz175, zzz176, zzz177, zzz178, zzz179, zzz18, zzz180, zzz181, zzz182, zzz183, zzz184, zzz185, zzz186, zzz187, zzz188, zzz189, zzz19, zzz190, zzz191, zzz192, zzz193, zzz194, zzz195, zzz196, zzz197, zzz198, zzz199, zzz2, zzz20, zzz200, zzz21, zzz22, zzz23, zzz24, zzz25, zzz26, zzz27, zzz28, zzz29, zzz3, zzz30, zzz31, zzz32, zzz33, zzz34, zzz35, zzz36, zzz37, zzz38, zzz39, zzz4, zzz40, zzz41, zzz42, zzz43, zzz44, zzz45, zzz46, zzz47, zzz48, zzz49, zzz5, zzz50, zzz51, zzz52, zzz53, zzz54, zzz55, zzz56, zzz57, zzz58, zzz59, zzz6, zzz60, zzz61, zzz62, zzz63, zzz64, zzz65, zzz66, zzz67, zzz68, zzz69, zzz7, zzz70, zzz71, zzz72, zzz73, zzz74, zzz75, zzz76, zzz77, zzz78, zzz79, zzz8, zzz80, zzz81, zzz82, zzz83, zzz84, zzz85, zzz86, zzz87, zzz88, zzz89, zzz9, zzz90, zzz91, zzz92, zzz93, zzz94, zzz95, zzz96, zzz97, zzz98, zzz99; ignoring DaemonSet-managed pods: tuned-fbvc2, dns-default-ggtfz, node-ca-d6hrd, machine-config-daemon-xhbhd, node-exporter-7wh4m, multus-t646v, ovs-vsgpp, sdn-7kbfm; deleting pods with local storage: alertmanager-main-0, grafana-867bfddd4d-rc8f5, prometheus-k8s-0 I0828 20:21:03.364673 3053 update.go:89] pod "prometheus-k8s-0" removed (evicted) I0828 20:22:04.362854 3053 update.go:89] pod "router-default-db4679547-xc77d" removed (evicted) I0828 20:22:04.964330 3053 update.go:89] pod "alertmanager-main-0" removed (evicted) I0828 20:22:46.363248 3053 update.go:89] pod "grafana-867bfddd4d-rc8f5" removed (evicted) $ oc logs -f -n openshift-machine-config-operator machine-config-daemon-qfvw7 I0828 20:26:22.866400 64123 start.go:68] Version: v4.2.0-201908271419-dirty (9e770a71b70d52651eb4c26abb0b9eb8d0b1a681) I0828 20:26:22.872373 64123 start.go:78] Calling chroot("/rootfs") I0828 20:26:22.872469 64123 rpm-ostree.go:356] Running captured: rpm-ostree status --json I0828 20:26:23.153010 64123 daemon.go:208] Booted osImageURL: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfaef42ab4bbd5826c43c3d1295638db1a6a4fe7d9cb9743145e4fa73b08e2fc (42.80.20190828.0) I0828 20:26:23.155826 64123 update.go:984] Starting to manage node: ip-10-0-133-89.us-west-2.compute.internal I0828 20:26:23.169764 64123 rpm-ostree.go:356] Running captured: rpm-ostree status I0828 20:26:23.356123 64123 daemon.go:715] State: idle AutomaticUpdates: disabled Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dfaef42ab4bbd5826c43c3d1295638db1a6a4fe7d9cb9743145e4fa73b08e2fc CustomOrigin: Managed by machine-config-operator Version: 42.80.20190828.0 (2019-08-28T04:59:49Z) pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86ddd9e4a84b171b55b09c368d4778cbb594c92f9ef95fd0e125c4f2dc7344c5 CustomOrigin: Image generated via coreos-assembler Version: 42.80.20190823.0 (2019-08-23T08:59:44Z) I0828 20:26:23.356144 64123 rpm-ostree.go:356] Running captured: journalctl --list-boots I0828 20:26:23.364447 64123 daemon.go:722] journalctl --list-boots: -1 7c014110075b4f2e8a705b3faabd6bfa Wed 2019-08-28 15:03:03 UTC—Wed 2019-08-28 15:06:15 UTC 0 a3775b7cd47b4fb2a27b6fecb5b695b1 Wed 2019-08-28 15:06:29 UTC—Wed 2019-08-28 20:26:23 UTC I0828 20:26:23.364472 64123 daemon.go:480] Starting MachineConfigDaemon I0828 20:26:23.364543 64123 daemon.go:487] Enabling Kubelet Healthz Monitor I0828 20:26:46.011064 64123 daemon.go:678] Current config: rendered-worker-94359b9747e9b078be0c2b0a0b04f7e2 I0828 20:26:46.011082 64123 daemon.go:679] Desired config: rendered-worker-aae690f6d10c4d045e1f3ad0da63865f I0828 20:26:46.011090 64123 update.go:984] drain interrupted, retrying I0828 20:26:46.014888 64123 update.go:984] Update prepared; beginning drain I0828 20:26:46.092826 64123 update.go:93] deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: ip-10-0-133-89us-west-2computeinternal-debug, zzz1, zzz10, zzz100, zzz101, zzz102, zzz103, zzz104, zzz105, zzz106, zzz107, zzz108, zzz109, zzz11, zzz110, zzz111, zzz112, zzz113, zzz114, zzz115, zzz116, zzz117, zzz118, zzz119, zzz12, zzz120, zzz121, zzz122, zzz123, zzz124, zzz125, zzz126, zzz127, zzz128, zzz129, zzz13, zzz130, zzz131, zzz132, zzz133, zzz134, zzz135, zzz136, zzz137, zzz138, zzz139, zzz14, zzz140, zzz141, zzz142, zzz143, zzz144, zzz145, zzz146, zzz147, zzz148, zzz149, zzz15, zzz150, zzz151, zzz152, zzz153, zzz154, zzz155, zzz156, zzz157, zzz158, zzz159, zzz16, zzz160, zzz161, zzz162, zzz163, zzz164, zzz165, zzz166, zzz167, zzz168, zzz169, zzz17, zzz170, zzz171, zzz172, zzz173, zzz174, zzz175, zzz176, zzz177, zzz178, zzz179, zzz18, zzz180, zzz181, zzz182, zzz183, zzz184, zzz185, zzz186, zzz187, zzz188, zzz189, zzz19, zzz190, zzz191, zzz192, zzz193, zzz194, zzz195, zzz196, zzz197, zzz198, zzz199, zzz2, zzz20, zzz200, zzz21, zzz22, zzz23, zzz24, zzz25, zzz26, zzz27, zzz28, zzz29, zzz3, zzz30, zzz31, zzz32, zzz33, zzz34, zzz35, zzz36, zzz37, zzz38, zzz39, zzz4, zzz40, zzz41, zzz42, zzz43, zzz44, zzz45, zzz46, zzz47, zzz48, zzz49, zzz5, zzz50, zzz51, zzz52, zzz53, zzz54, zzz55, zzz56, zzz57, zzz58, zzz59, zzz6, zzz60, zzz61, zzz62, zzz63, zzz64, zzz65, zzz66, zzz67, zzz68, zzz69, zzz7, zzz70, zzz71, zzz72, zzz73, zzz74, zzz75, zzz76, zzz77, zzz78, zzz79, zzz8, zzz80, zzz81, zzz82, zzz83, zzz84, zzz85, zzz86, zzz87, zzz88, zzz89, zzz9, zzz90, zzz91, zzz92, zzz93, zzz94, zzz95, zzz96, zzz97, zzz98, zzz99; ignoring DaemonSet-managed pods: tuned-fbvc2, dns-default-ggtfz, node-ca-d6hrd, machine-config-daemon-qfvw7, node-exporter-7wh4m, multus-t646v, ovs-vsgpp, sdn-7kbfm I0828 20:31:08.621806 64123 update.go:89] pod "zzz101" removed (evicted) I0828 20:31:30.626156 64123 update.go:89] pod "zzz103" removed (evicted) I0828 20:31:33.425777 64123 update.go:89] pod "zzz104" removed (evicted) I0828 20:31:34.018977 64123 update.go:89] pod "zzz106" removed (evicted) I0828 20:31:56.626549 64123 update.go:89] pod "zzz117" removed (evicted) I0828 20:32:04.222172 64123 update.go:89] pod "zzz102" removed (evicted) I0828 20:32:04.618342 64123 update.go:89] pod "zzz1" removed (evicted) I0828 20:32:05.017889 64123 update.go:89] pod "zzz10" removed (evicted) I0828 20:32:06.418995 64123 update.go:89] pod "zzz12" removed (evicted) I0828 20:32:13.418038 64123 update.go:89] pod "zzz105" removed (evicted) I0828 20:32:17.217839 64123 update.go:89] pod "zzz107" removed (evicted) I0828 20:32:18.617535 64123 update.go:89] pod "zzz180" removed (evicted) I0828 20:32:18.817578 64123 update.go:89] pod "zzz142" removed (evicted) I0828 20:32:20.617793 64123 update.go:89] pod "zzz143" removed (evicted) I0828 20:32:25.017645 64123 update.go:89] pod "zzz40" removed (evicted) I0828 20:32:28.017626 64123 update.go:89] pod "zzz108" removed (evicted) I0828 20:32:28.417756 64123 update.go:89] pod "zzz109" removed (evicted) I0828 20:32:29.617834 64123 update.go:89] pod "zzz43" removed (evicted) I0828 20:32:31.618070 64123 update.go:89] pod "zzz15" removed (evicted) I0828 20:32:32.017777 64123 update.go:89] pod "zzz149" removed (evicted) I0828 20:32:32.417989 64123 update.go:89] pod "zzz7" removed (evicted) I0828 20:32:32.818004 64123 update.go:89] pod "zzz45" removed (evicted) I0828 20:32:33.417832 64123 update.go:89] pod "zzz70" removed (evicted) I0828 20:32:33.617797 64123 update.go:89] pod "zzz46" removed (evicted) I0828 20:32:37.417830 64123 update.go:89] pod "zzz118" removed (evicted) I0828 20:32:39.217843 64123 update.go:89] pod "zzz28" removed (evicted) I0828 20:32:41.818255 64123 update.go:89] pod "zzz155" removed (evicted) I0828 20:32:42.217724 64123 update.go:89] pod "zzz51" removed (evicted) I0828 20:32:43.418022 64123 update.go:89] pod "zzz100" removed (evicted) I0828 20:32:44.417793 64123 update.go:89] pod "zzz157" removed (evicted) I0828 20:32:44.618006 64123 update.go:89] pod "zzz53" removed (evicted) I0828 20:32:44.817824 64123 update.go:89] pod "zzz158" removed (evicted) I0828 20:32:45.217864 64123 update.go:89] pod "zzz3" removed (evicted) I0828 20:32:45.617816 64123 update.go:89] pod "zzz159" removed (evicted) I0828 20:32:46.017668 64123 update.go:89] pod "zzz56" removed (evicted) I0828 20:32:46.617878 64123 update.go:89] pod "zzz136" removed (evicted) I0828 20:32:47.217845 64123 update.go:89] pod "zzz120" removed (evicted) I0828 20:32:47.617888 64123 update.go:89] pod "zzz160" removed (evicted) I0828 20:32:48.017676 64123 update.go:89] pod "zzz32" removed (evicted) I0828 20:32:48.417949 64123 update.go:89] pod "zzz123" removed (evicted) I0828 20:32:48.617833 64123 update.go:89] pod "zzz59" removed (evicted) I0828 20:32:48.817798 64123 update.go:89] pod "zzz138" removed (evicted) I0828 20:32:49.017912 64123 update.go:89] pod "zzz163" removed (evicted) I0828 20:32:49.217616 64123 update.go:89] pod "zzz137" removed (evicted) I0828 20:32:49.417928 64123 update.go:89] pod "zzz161" removed (evicted) I0828 20:32:50.217676 64123 update.go:89] pod "zzz124" removed (evicted) I0828 20:32:50.617824 64123 update.go:89] pod "zzz33" removed (evicted) I0828 20:32:51.617693 64123 update.go:89] pod "zzz34" removed (evicted) I0828 20:32:52.018028 64123 update.go:89] pod "zzz139" removed (evicted) I0828 20:32:53.217612 64123 update.go:89] pod "zzz111" removed (evicted) I0828 20:32:53.418037 64123 update.go:89] pod "zzz173" removed (evicted) I0828 20:32:54.217612 64123 update.go:89] pod "zzz174" removed (evicted) I0828 20:32:54.617871 64123 update.go:89] pod "zzz177" removed (evicted) I0828 20:32:55.819792 64123 update.go:89] pod "zzz38" removed (evicted) I0828 20:32:56.017723 64123 update.go:89] pod "zzz141" removed (evicted) I0828 20:32:57.017880 64123 update.go:89] pod "zzz112" removed (evicted) I0828 20:32:58.017983 64123 update.go:89] pod "zzz184" removed (evicted) I0828 20:32:59.217743 64123 update.go:89] pod "zzz39" removed (evicted) I0828 20:32:59.617859 64123 update.go:89] pod "zzz4" removed (evicted) I0828 20:33:00.217840 64123 update.go:89] pod "zzz144" removed (evicted) I0828 20:33:00.817686 64123 update.go:89] pod "zzz128" removed (evicted) I0828 20:33:01.818096 64123 update.go:89] pod "zzz145" removed (evicted) I0828 20:33:03.217756 64123 update.go:89] pod "zzz41" removed (evicted) I0828 20:33:03.417792 64123 update.go:89] pod "zzz146" removed (evicted) I0828 20:33:04.817811 64123 update.go:89] pod "zzz42" removed (evicted) I0828 20:33:07.217810 64123 update.go:89] pod "zzz115" removed (evicted) I0828 20:33:08.017925 64123 update.go:89] pod "zzz67" removed (evicted) I0828 20:33:08.617630 64123 update.go:89] pod "zzz130" removed (evicted) I0828 20:33:09.217694 64123 update.go:89] pod "zzz47" removed (evicted) I0828 20:33:09.817983 64123 update.go:89] pod "zzz73" removed (evicted) I0828 20:33:10.018100 64123 update.go:89] pod "zzz116" removed (evicted) I0828 20:33:10.217428 64123 update.go:89] pod "zzz150" removed (evicted) I0828 20:33:12.217805 64123 update.go:89] pod "zzz132" removed (evicted) I0828 20:33:12.418165 64123 update.go:89] pod "zzz49" removed (evicted) I0828 20:33:14.017783 64123 update.go:89] pod "zzz119" removed (evicted) I0828 20:33:15.018977 64123 update.go:89] pod "zzz5" removed (evicted) I0828 20:33:15.417998 64123 update.go:89] pod "zzz86" removed (evicted) I0828 20:33:16.419331 64123 update.go:89] pod "zzz154" removed (evicted) I0828 20:33:18.817779 64123 update.go:89] pod "zzz134" removed (evicted) I0828 20:33:19.417825 64123 update.go:89] pod "zzz30" removed (evicted) I0828 20:33:20.417732 64123 update.go:89] pod "zzz121" removed (evicted) I0828 20:33:20.617815 64123 update.go:89] pod "zzz58" removed (evicted) I0828 20:33:21.017726 64123 update.go:89] pod "zzz164" removed (evicted) I0828 20:33:21.617901 64123 update.go:89] pod "zzz166" removed (evicted) I0828 20:33:22.618109 64123 update.go:89] pod "zzz170" removed (evicted) I0828 20:33:22.817975 64123 update.go:89] pod "zzz171" removed (evicted) I0828 20:33:23.017726 64123 update.go:89] pod "zzz14" removed (evicted) I0828 20:33:24.617989 64123 update.go:89] pod "zzz18" removed (evicted) I0828 20:33:24.817781 64123 update.go:89] pod "zzz36" removed (evicted) I0828 20:33:25.217798 64123 update.go:89] pod "zzz21" removed (evicted) I0828 20:33:25.617808 64123 update.go:89] pod "zzz37" removed (evicted) I0828 20:33:27.417749 64123 update.go:89] pod "zzz127" removed (evicted) I0828 20:33:27.617745 64123 update.go:89] pod "zzz187" removed (evicted) I0828 20:33:28.417949 64123 update.go:89] pod "zzz190" removed (evicted) I0828 20:33:28.817802 64123 update.go:89] pod "zzz191" removed (evicted) I0828 20:33:29.017733 64123 update.go:89] pod "zzz192" removed (evicted) I0828 20:33:29.218022 64123 update.go:89] pod "zzz194" removed (evicted) I0828 20:33:29.417896 64123 update.go:89] pod "zzz23" removed (evicted) I0828 20:33:31.417881 64123 update.go:89] pod "zzz200" removed (evicted) I0828 20:33:32.617837 64123 update.go:89] pod "zzz62" removed (evicted) I0828 20:33:33.217779 64123 update.go:89] pod "zzz13" removed (evicted) I0828 20:33:33.817944 64123 update.go:89] pod "zzz64" removed (evicted) I0828 20:33:34.217908 64123 update.go:89] pod "zzz65" removed (evicted) I0828 20:33:34.417694 64123 update.go:89] pod "zzz66" removed (evicted) I0828 20:33:34.617886 64123 update.go:89] pod "zzz69" removed (evicted) I0828 20:33:34.817509 64123 update.go:89] pod "zzz25" removed (evicted) I0828 20:33:36.417986 64123 update.go:89] pod "zzz151" removed (evicted) I0828 20:33:36.817863 64123 update.go:89] pod "zzz131" removed (evicted) I0828 20:33:37.418201 64123 update.go:89] pod "zzz77" removed (evicted) I0828 20:33:37.617878 64123 update.go:89] pod "zzz27" removed (evicted) I0828 20:33:38.817728 64123 update.go:89] pod "zzz82" removed (evicted) I0828 20:33:39.217907 64123 update.go:89] pod "zzz153" removed (evicted) I0828 20:33:39.817678 64123 update.go:89] pod "zzz29" removed (evicted) I0828 20:33:40.017907 64123 update.go:89] pod "zzz87" removed (evicted) I0828 20:33:40.217939 64123 update.go:89] pod "zzz9" removed (evicted) I0828 20:33:40.417951 64123 update.go:89] pod "zzz133" removed (evicted) I0828 20:33:40.617907 64123 update.go:89] pod "zzz90" removed (evicted) I0828 20:33:41.017845 64123 update.go:89] pod "zzz91" removed (evicted) I0828 20:33:41.617760 64123 update.go:89] pod "zzz92" removed (evicted) I0828 20:33:43.017996 64123 update.go:89] pod "zzz52" removed (evicted) I0828 20:33:43.417823 64123 update.go:89] pod "zzz55" removed (evicted) I0828 20:33:43.817720 64123 update.go:89] pod "zzz122" removed (evicted) I0828 20:33:45.217843 64123 update.go:89] pod "zzz110" removed (evicted) I0828 20:33:45.417782 64123 update.go:89] pod "zzz169" removed (evicted) I0828 20:33:45.817948 64123 update.go:89] pod "zzz17" removed (evicted) I0828 20:33:46.217876 64123 update.go:89] pod "zzz172" removed (evicted) I0828 20:33:46.417790 64123 update.go:89] pod "zzz125" removed (evicted) I0828 20:33:47.017800 64123 update.go:89] pod "zzz126" removed (evicted) I0828 20:33:47.817872 64123 update.go:89] pod "zzz185" removed (evicted) I0828 20:33:48.017858 64123 update.go:89] pod "zzz179" removed (evicted) I0828 20:33:49.417629 64123 update.go:89] pod "zzz189" removed (evicted) I0828 20:33:50.017738 64123 update.go:89] pod "zzz114" removed (evicted) I0828 20:33:50.217890 64123 update.go:89] pod "zzz195" removed (evicted) I0828 20:33:50.818041 64123 update.go:89] pod "zzz198" removed (evicted) I0828 20:33:51.617744 64123 update.go:89] pod "zzz20" removed (evicted) I0828 20:33:51.817788 64123 update.go:89] pod "zzz199" removed (evicted) I0828 20:33:52.417753 64123 update.go:89] pod "zzz2" removed (evicted) I0828 20:33:52.817777 64123 update.go:89] pod "zzz63" removed (evicted) I0828 20:33:53.017706 64123 update.go:89] pod "zzz61" removed (evicted) I0828 20:33:53.817990 64123 update.go:89] pod "zzz44" removed (evicted) I0828 20:33:54.217776 64123 update.go:89] pod "zzz68" removed (evicted) I0828 20:33:54.417796 64123 update.go:89] pod "zzz72" removed (evicted) I0828 20:33:55.417683 64123 update.go:89] pod "zzz78" removed (evicted) I0828 20:33:56.017877 64123 update.go:89] pod "zzz81" removed (evicted) I0828 20:33:56.217963 64123 update.go:89] pod "zzz152" removed (evicted) I0828 20:33:56.617842 64123 update.go:89] pod "zzz48" removed (evicted) I0828 20:33:57.617750 64123 update.go:89] pod "zzz50" removed (evicted) I0828 20:33:57.817765 64123 update.go:89] pod "zzz156" removed (evicted) I0828 20:33:58.217789 64123 update.go:89] pod "zzz93" removed (evicted) I0828 20:33:58.417617 64123 update.go:89] pod "zzz94" removed (evicted) I0828 20:33:58.817803 64123 update.go:89] pod "zzz96" removed (evicted) I0828 20:33:59.617931 64123 update.go:89] pod "zzz135" removed (evicted) I0828 20:34:00.217689 64123 update.go:89] pod "zzz31" removed (evicted) I0828 20:34:01.018030 64123 update.go:89] pod "zzz168" removed (evicted) I0828 20:34:01.817992 64123 update.go:89] pod "zzz140" removed (evicted) I0828 20:34:02.017783 64123 update.go:89] pod "zzz181" removed (evicted) I0828 20:34:02.418011 64123 update.go:89] pod "zzz22" removed (evicted) I0828 20:34:04.417811 64123 update.go:89] pod "zzz196" removed (evicted) I0828 20:34:04.617643 64123 update.go:89] pod "zzz24" removed (evicted) I0828 20:34:04.817884 64123 update.go:89] pod "zzz6" removed (evicted) I0828 20:34:05.617819 64123 update.go:89] pod "zzz11" removed (evicted) I0828 20:34:05.817979 64123 update.go:89] pod "zzz60" removed (evicted) I0828 20:34:06.017750 64123 update.go:89] pod "zzz148" removed (evicted) I0828 20:34:06.217807 64123 update.go:89] pod "zzz71" removed (evicted) I0828 20:34:06.417904 64123 update.go:89] pod "zzz26" removed (evicted) I0828 20:34:06.818060 64123 update.go:89] pod "zzz76" removed (evicted) I0828 20:34:07.017775 64123 update.go:89] pod "zzz74" removed (evicted) I0828 20:34:07.217878 64123 update.go:89] pod "zzz89" removed (evicted) I0828 20:34:08.617813 64123 update.go:89] pod "zzz54" removed (evicted) I0828 20:34:09.617795 64123 update.go:89] pod "zzz57" removed (evicted) I0828 20:34:10.017848 64123 update.go:89] pod "zzz167" removed (evicted) I0828 20:34:10.217788 64123 update.go:89] pod "zzz162" removed (evicted) I0828 20:34:10.617680 64123 update.go:89] pod "zzz175" removed (evicted) I0828 20:34:11.017927 64123 update.go:89] pod "zzz176" removed (evicted) I0828 20:34:11.217808 64123 update.go:89] pod "zzz183" removed (evicted) I0828 20:34:11.417730 64123 update.go:89] pod "zzz178" removed (evicted) I0828 20:34:12.017850 64123 update.go:89] pod "zzz113" removed (evicted) I0828 20:34:12.217752 64123 update.go:89] pod "zzz186" removed (evicted) I0828 20:34:12.617768 64123 update.go:89] pod "zzz129" removed (evicted) I0828 20:34:13.217795 64123 update.go:89] pod "zzz79" removed (evicted) I0828 20:34:13.617804 64123 update.go:89] pod "zzz99" removed (evicted) I0828 20:34:13.817667 64123 update.go:89] pod "zzz75" removed (evicted) I0828 20:34:14.217706 64123 update.go:89] pod "zzz83" removed (evicted) I0828 20:34:14.817867 64123 update.go:89] pod "zzz88" removed (evicted) I0828 20:34:15.417839 64123 update.go:89] pod "zzz95" removed (evicted) I0828 20:34:16.217876 64123 update.go:89] pod "zzz165" removed (evicted) I0828 20:34:16.817933 64123 update.go:89] pod "zzz188" removed (evicted) I0828 20:34:17.217792 64123 update.go:89] pod "zzz193" removed (evicted) I0828 20:34:17.417860 64123 update.go:89] pod "zzz147" removed (evicted) I0828 20:34:17.617822 64123 update.go:89] pod "zzz197" removed (evicted) I0828 20:34:17.817832 64123 update.go:89] pod "zzz80" removed (evicted) I0828 20:34:18.018079 64123 update.go:89] pod "zzz84" removed (evicted) I0828 20:34:18.417789 64123 update.go:89] pod "zzz8" removed (evicted) I0828 20:34:19.817915 64123 update.go:89] pod "zzz19" removed (evicted) I0828 20:34:20.217662 64123 update.go:89] pod "zzz16" removed (evicted) I0828 20:34:20.817735 64123 update.go:89] pod "zzz98" removed (evicted) I0828 20:34:21.018029 64123 update.go:89] pod "zzz182" removed (evicted) I0828 20:34:21.217749 64123 update.go:89] pod "zzz35" removed (evicted) I0828 20:34:21.617582 64123 update.go:89] pod "zzz97" removed (evicted) I0828 20:34:22.420833 64123 update.go:89] pod "zzz85" removed (evicted) I0828 20:36:53.116029 64123 update.go:89] pod "ip-10-0-133-89us-west-2computeinternal-debug" removed (evicted) I0828 20:36:53.116060 64123 update.go:89] drained node "ip-10-0-133-89.us-west-2.compute.internal" I0828 20:36:53.116069 64123 update.go:984] drain complete I0828 20:36:53.118677 64123 update.go:984] initiating reboot: Node will reboot into config rendered-worker-aae690f6d10c4d045e1f3ad0da63865f
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922