Description of problem: Customer reports fluentd pods on the master nodes keep restarting. Open inspection of the master node journals, the atomic-openshift-node is periodically stopping and restarting and as a result, they are seeing failures of the fluentd daemon set. Version-Release number of selected component (if applicable): v3.11.43 How reproducible: Consistently - all master nodes are impacted Steps to Reproduce: 1. Start master 2. Node will periodically restart every 30-90 minutes Actual results: Node service is periodically restarting Expected results: Nodes service should not be restarting Additional info: - Excerpt of events from openshift-logging showing failing daemon set - NAME READY STATUS RESTARTS AGE logging-curator-1556767800-tzqdk 0/1 Error 0 11h logging-es-data-master-jbcznqy0-1-nkxn9 2/2 Running 5901 30d logging-fluentd-5crnp 1/1 Running 1 29d logging-fluentd-9n77p 1/1 Running 2 146d logging-fluentd-bv2bb 1/1 Running 3 146d logging-fluentd-jqqnb 1/1 Running 0 46m logging-fluentd-mbp7r 1/1 Running 1 26d logging-fluentd-nfqjk 1/1 Running 0 1h logging-fluentd-pqnqq 1/1 Running 0 7h logging-fluentd-zb5w8 1/1 Running 2 98d logging-fluentd-zwzjs 1/1 Running 2 146d logging-kibana-1-w7kkc 2/2 Running 4 131d [cloud-user@ip-10-x-x-22 ~]$ oc get events --sort-by=.metadata.creationTimestamp | grep failed | grep Warning | tail -n 12 1h 1h 1 logging-fluentd.159ae0d9b065cfdb DaemonSet Warning FailedDaemonPod daemonset-controller Found failed daemon pod openshift-logging/logging-fluentd-qpf6w on node ip-10-x-x-79.ec2.internal, will try to kill it 1h 1h 1 logging-fluentd.159ae0d9d4263bdd DaemonSet Warning FailedDaemonPod daemonset-controller Found failed daemon pod openshift-logging/logging-fluentd-rjnhj on node ip-10-x-x-79.ec2.internal, will try to kill it 1h 1h 1 logging-fluentd-tczjf.159ae0d9d681ab47 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-79.ec2.internal Predicate MatchNodeSelector failed 1h 1h 1 logging-fluentd.159ae0d9f7f0a5e0 DaemonSet Warning FailedDaemonPod daemonset-controller Found failed daemon pod openshift-logging/logging-fluentd-tczjf on node ip-10-x-x-79.ec2.internal, will try to kill it 1h 1h 1 logging-fluentd-xpljx.159ae0da1dcde0a0 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-79.ec2.internal Predicate MatchNodeSelector failed 1h 1h 1 logging-fluentd-4llwt.159ae0d9fa2a9b25 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-79.ec2.internal Predicate MatchNodeSelector failed 1h 1h 1 logging-fluentd-pmldv.159ae0da6569b3d3 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-79.ec2.internal Predicate MatchNodeSelector failed 1h 1h 1 logging-fluentd-6h5nc.159ae0da418d3ee6 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-79.ec2.internal Predicate MatchNodeSelector failed 44m 44m 1 logging-fluentd-bh9nd.159ae29708206ac1 Pod Warning MatchNodeSelector kubelet, ip-10-x-x-37.ec2.internal Predicate MatchNodeSelector failed 44m 44m 1 logging-fluentd-vjjs2.159ae298c14ec6ec Pod Warning MatchNodeSelector kubelet, ip-10-x-x-37.ec2.int - Excerpt from journal of impacted master node - abr 21 15:49:03 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:03.735875 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: setting upstream servers from DBus abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 10.35.2.2#53 abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain in-addr.arpa abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain cluster.local abr 21 15:49:13 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:13.735990 504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" abr 21 15:49:13 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:13.739797 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:23 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:23.739923 504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" abr 21 15:49:23 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:23.743013 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:33 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:33.743130 504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" abr 21 15:49:33 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:33.746485 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: setting upstream servers from DBus abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 10.35.2.2#53 abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain in-addr.arpa abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain cluster.local abr 21 15:49:43 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:43.746627 504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" abr 21 15:49:43 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:43.750004 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:53 ip-10-x-x-37.ec2.internal oci-systemd-hook[80286]: systemdhook <debug>: 1da5147dd953: Skipping as container command is /bin/bash, not init or systemd abr 21 15:49:53 ip-10-x-x-37.ec2.internal oci-umount[80287]: umounthook <debug>: 1da5147dd953: only runs in prestart stage, ignoring abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:53 ip-10-x-x-37.ec2.internal dockerd-current[6774]: time="2019-04-21T15:49:53.287080934-05:00" level=warning msg="1da5147dd953d1785620551c9f9bec70673e86e5ca352cd6f150857a55eb63d1 cleanup: failed to unmount secrets: invalid argument" abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.404985 504 kubelet.go:1865] SyncLoop (PLEG): "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)", event: &pleg.PodLifecycleEvent{ID:"b0d98acc-f9da-11e8-9d54-0ab51f65d96e", Type:"ContainerDied", Data:"1da5147dd953d1785620551c9f9bec70673e86e5ca352cd6f150857a55eb63d1"} abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.713996 504 kuberuntime_manager.go:513] Container {Name:sync Image:registry.redhat.io/openshift3/ose-node:v3.11 Command:[/bin/bash -c #!/bin/bash abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: set -euo pipefail abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # set by the node image abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: unset KUBECONFIG abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: trap 'kill $(jobs -p); exit 0' TERM abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # track the current state of the config abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -f /etc/origin/node/node-config.yaml ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: md5sum /etc/origin/node/node-config.yaml > /tmp/.old abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: touch /tmp/.old abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # loop until BOOTSTRAP_CONFIG_NAME is set abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/origin-node abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -f /etc/sysconfig/atomic-openshift-node ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/atomic-openshift-node abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: elif [[ -f /etc/sysconfig/origin-node ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/origin-node abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Waiting for the node sysconfig file to be created" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 & wait abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: name="$(sed -nE 's|^BOOTSTRAP_CONFIG_NAME=([^#].+)|\1|p' "${file}" | head -1)" abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -z "${name}" ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Waiting for BOOTSTRAP_CONFIG_NAME to be set" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 & wait abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # in the background check to see if the value changes and exit if so abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: pid=$BASHPID abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: ( abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! updated="$(sed -nE 's|^BOOTSTRAP_CONFIG_NAME=([^#].+)|\1|p' "${file}" | head -1)"; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to check for bootstrap config, exiting" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: kill $pid abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: exit 1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ "${updated}" != "${name}" ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Bootstrap configuration profile name changed, exiting" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: kill $pid abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: exit 0 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: ) & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: break abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: mkdir -p /etc/origin/node/tmp abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # periodically refresh both node-config.yaml and relabel the node abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! oc extract "configmaps/${name}" -n openshift-node --to=/etc/origin/node/tmp --confirm --request-timeout=10s --config /etc/origin/node/node.kubeconfig "--token=$( cat /var/run/secrets/kubernetes.io/serviceaccount/token )" > /dev/null; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to retrieve latest config for node" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $! abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: KUBELET_HOSTNAME_OVERRIDE=$(cat /etc/sysconfig/KUBELET_HOSTNAME_OVERRIDE) || : abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! [[ -z "$KUBELET_HOSTNAME_OVERRIDE" ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: #Patching node-config for hostname override abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "nodeName: $KUBELET_HOSTNAME_OVERRIDE" >> /etc/origin/node/tmp/node-config.yaml abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # detect whether the node-config.yaml has changed, and if so trigger a restart of the kubelet. abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: md5sum /etc/origin/node/tmp/node-config.yaml > /tmp/.new abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ "$( cat /tmp/.old )" != "$( cat /tmp/.new )" ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: mv /etc/origin/node/tmp/node-config.yaml /etc/origin/node/node-config.yaml abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: SYSTEMD_IGNORE_CHROOT=1 systemctl restart tuned || : abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Configuration changed, restarting kubelet" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # TODO: kubelet doesn't relabel nodes, best effort for now abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # https://github.com/kubernetes/kubernetes/issues/59314 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if args="$(openshift-node-config --config /etc/origin/node/node-config.yaml)"; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: labels=$(tr ' ' '\n' <<<$args | sed -ne '/^--node-labels=/ { s/^--node-labels=//; p; }' | tr ',\n' ' ') abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -n "${labels}" ]]; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Applying node labels $labels" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! oc label --config=/etc/origin/node/node.kubeconfig "node/${NODE_NAME}" ${labels} --overwrite; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to apply labels, will retry in 10" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $! abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: The downloaded node configuration is invalid, retrying later" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $! abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! pkill -U 0 -f '(^|/)hyperkube kubelet '; then abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to restart Kubelet" 2>&1 abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $! abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # annotate node with md5sum of the config abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: oc annotate --config=/etc/origin/node/node.kubeconfig "node/${NODE_NAME}" \ abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: node.openshift.io/md5sum="$( cat /tmp/.new | cut -d' ' -f1 )" --overwrite abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: cp -f /tmp/.new /tmp/.old abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 180 & abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $! abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: ] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:host-config ReadOnly:false MountPath:/etc/origin/node/ SubPath: MountPropagation:<nil>} {Name:host-sysconfig-node ReadOnly:true MountPath:/etc/sysconfig SubPath: MountPropagation:<nil>} {Name:var-run-dbus ReadOnly:true MountPath:/var/run/dbus SubPath: MountPropagation:<nil>} {Name:run-systemd-system ReadOnly:true MountPath:/run/systemd/system SubPath: MountPropagation:<nil>} {Name:sync-token-g75k9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.714272 504 kuberuntime_manager.go:757] checking backoff for container "sync" in pod "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)" abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.750145 504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.753161 504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent) abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount abr 21 15:49:54 ip-10-x-x-37.ec2.internal systemd[1]: Started libcontainer container f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c. -- Subject: Unit docker-f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c.scope has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit docker-f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c.scope has finished starting up. -- -- The start-up result is done. abr 21 15:49:54 ip-10-x-x-37.ec2.internal oci-systemd-hook[80422]: systemdhook <debug>: f9365a3ea4b0: Skipping as container command is /bin/bash, not init or systemd abr 21 15:49:54 ip-10-x-x-37.ec2.internal oci-umount[80423]: umounthook <debug>: prestart container_id:f9365a3ea4b0 rootfs:/var/lib/docker/devicemapper/mnt/9b4b9d6d74390bb00384017d66f53cf0d47637a72bc9d527bab7d26e395ade72/rootfs abr 21 15:49:54 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:54.635000 504 kubelet.go:1865] SyncLoop (PLEG): "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)", event: &pleg.PodLifecycleEvent{ID:"b0d98acc-f9da-11e8-9d54-0ab51f65d96e", Type:"ContainerStarted", Data:"f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c"} abr 21 15:49:58 ip-10-x-x-37.ec2.internal systemd[1]: Stopping Dynamic System Tuning Daemon... -- Subject: Unit tuned.service has begun shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit tuned.service has begun shutting down. abr 21 15:49:59 ip-10-x-x-37.ec2.internal kernel: nf_conntrack: falling back to vmalloc. abr 21 15:49:59 ip-10-x-x-37.ec2.internal systemd[1]: Stopped Dynamic System Tuning Daemon. -- Subject: Unit tuned.service has finished shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit tuned.service has finished shutting down. abr 21 15:49:59 ip-10-x-x-37.ec2.internal systemd[1]: Starting Dynamic System Tuning Daemon... -- Subject: Unit tuned.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit tuned.service has begun starting up. abr 21 15:50:00 ip-10-x-x-37.ec2.internal kernel: nf_conntrack: falling back to vmalloc. abr 21 15:50:00 ip-10-x-x-37.ec2.internal systemd[1]: Started Dynamic System Tuning Daemon. -- Subject: Unit tuned.service has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit tuned.service has finished starting up. -- -- The start-up result is done. abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: atomic-openshift-node.service holdoff time over, scheduling restart. abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: Stopped OpenShift Node. -- Subject: Unit atomic-openshift-node.service has finished shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit atomic-openshift-node.service has finished shutting down. abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: Starting OpenShift Node... -- Subject: Unit atomic-openshift-node.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit atomic-openshift-node.service has begun starting up. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --allow-privileged has been deprecated, will be removed in a future version abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authentication-token-webhook-cache-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-webhook-cache-authorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-webhook-cache-unauthorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.11, and the cadvisor port will be removed entirely in 1.12 abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
If there are any suggestions of things to look at or additional diagnostic information I can gather, I would really appreciate it.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2477