Bug 1731236 - atomic-openshift-node.service keeps stopping and starting several times a day
Summary: atomic-openshift-node.service keeps stopping and starting several times a day
Keywords:
Status: POST
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.11.z
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-18 17:42 UTC by rvanderp
Modified: 2019-11-07 16:43 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift openshift-tools pull 4256 None None None 2019-11-07 16:43:29 UTC

Description rvanderp 2019-07-18 17:42:50 UTC
Description of problem:
Customer reports fluentd pods on the master nodes keep restarting.  Open inspection of the master node journals, the atomic-openshift-node is periodically stopping and restarting and as a result, they are seeing failures of the fluentd daemon set.

Version-Release number of selected component (if applicable):
v3.11.43

How reproducible:
Consistently - all master nodes are impacted

Steps to Reproduce:
1. Start master 
2. Node will periodically restart every 30-90 minutes


Actual results:

Node service is periodically restarting

Expected results:

Nodes service should not be restarting


Additional info:

- Excerpt of events from openshift-logging showing failing daemon set - 

NAME                                      READY     STATUS    RESTARTS   AGE
logging-curator-1556767800-tzqdk          0/1       Error     0          11h
logging-es-data-master-jbcznqy0-1-nkxn9   2/2       Running   5901       30d
logging-fluentd-5crnp                     1/1       Running   1          29d
logging-fluentd-9n77p                     1/1       Running   2          146d
logging-fluentd-bv2bb                     1/1       Running   3          146d
logging-fluentd-jqqnb                     1/1       Running   0          46m
logging-fluentd-mbp7r                     1/1       Running   1          26d
logging-fluentd-nfqjk                     1/1       Running   0          1h
logging-fluentd-pqnqq                     1/1       Running   0          7h
logging-fluentd-zb5w8                     1/1       Running   2          98d
logging-fluentd-zwzjs                     1/1       Running   2          146d
logging-kibana-1-w7kkc                    2/2       Running   4          131d
[cloud-user@ip-10-x-x-22 ~]$ oc get events --sort-by=.metadata.creationTimestamp | grep failed | grep Warning | tail -n 12
1h          1h           1         logging-fluentd.159ae0d9b065cfdb         DaemonSet                                            Warning   FailedDaemonPod     daemonset-controller                  Found failed daemon pod openshift-logging/logging-fluentd-qpf6w on node ip-10-x-x-79.ec2.internal, will try to kill it
1h          1h           1         logging-fluentd.159ae0d9d4263bdd         DaemonSet                                            Warning   FailedDaemonPod     daemonset-controller                  Found failed daemon pod openshift-logging/logging-fluentd-rjnhj on node ip-10-x-x-79.ec2.internal, will try to kill it
1h          1h           1         logging-fluentd-tczjf.159ae0d9d681ab47   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-79.ec2.internal   Predicate MatchNodeSelector failed
1h          1h           1         logging-fluentd.159ae0d9f7f0a5e0         DaemonSet                                            Warning   FailedDaemonPod     daemonset-controller                  Found failed daemon pod openshift-logging/logging-fluentd-tczjf on node ip-10-x-x-79.ec2.internal, will try to kill it
1h          1h           1         logging-fluentd-xpljx.159ae0da1dcde0a0   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-79.ec2.internal   Predicate MatchNodeSelector failed
1h          1h           1         logging-fluentd-4llwt.159ae0d9fa2a9b25   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-79.ec2.internal   Predicate MatchNodeSelector failed
1h          1h           1         logging-fluentd-pmldv.159ae0da6569b3d3   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-79.ec2.internal   Predicate MatchNodeSelector failed
1h          1h           1         logging-fluentd-6h5nc.159ae0da418d3ee6   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-79.ec2.internal   Predicate MatchNodeSelector failed
44m         44m          1         logging-fluentd-bh9nd.159ae29708206ac1   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-37.ec2.internal   Predicate MatchNodeSelector failed
44m         44m          1         logging-fluentd-vjjs2.159ae298c14ec6ec   Pod                                                  Warning   MatchNodeSelector   kubelet, ip-10-x-x-37.ec2.int






- Excerpt from journal of impacted master node - 

abr 21 15:49:03 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:03.735875     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: setting upstream servers from DBus
abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 10.35.2.2#53
abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
abr 21 15:49:07 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain cluster.local
abr 21 15:49:13 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:13.735990     504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal"
abr 21 15:49:13 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:13.739797     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:23 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:23.739923     504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal"
abr 21 15:49:23 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:23.743013     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:33 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:33.743130     504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal"
abr 21 15:49:33 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:33.746485     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: setting upstream servers from DBus
abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 10.35.2.2#53
abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain in-addr.arpa
abr 21 15:49:37 ip-10-x-x-37.ec2.internal dnsmasq[3683]: using nameserver 127.0.0.1#53 for domain cluster.local
abr 21 15:49:43 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:43.746627     504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal"
abr 21 15:49:43 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:43.750004     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:53 ip-10-x-x-37.ec2.internal oci-systemd-hook[80286]: systemdhook <debug>: 1da5147dd953: Skipping as container command is /bin/bash, not init or systemd
abr 21 15:49:53 ip-10-x-x-37.ec2.internal oci-umount[80287]: umounthook <debug>: 1da5147dd953: only runs in prestart stage, ignoring
abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem
abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove
abr 21 15:49:53 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:53 ip-10-x-x-37.ec2.internal dockerd-current[6774]: time="2019-04-21T15:49:53.287080934-05:00" level=warning msg="1da5147dd953d1785620551c9f9bec70673e86e5ca352cd6f150857a55eb63d1 cleanup: failed to unmount secrets: invalid argument"
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.404985     504 kubelet.go:1865] SyncLoop (PLEG): "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)", event: &pleg.PodLifecycleEvent{ID:"b0d98acc-f9da-11e8-9d54-0ab51f65d96e", Type:"ContainerDied", Data:"1da5147dd953d1785620551c9f9bec70673e86e5ca352cd6f150857a55eb63d1"}
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.713996     504 kuberuntime_manager.go:513] Container {Name:sync Image:registry.redhat.io/openshift3/ose-node:v3.11 Command:[/bin/bash -c #!/bin/bash
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: set -euo pipefail
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # set by the node image
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: unset KUBECONFIG
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: trap 'kill $(jobs -p); exit 0' TERM
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # track the current state of the config
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -f /etc/origin/node/node-config.yaml ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: md5sum /etc/origin/node/node-config.yaml > /tmp/.old
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: touch /tmp/.old
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # loop until BOOTSTRAP_CONFIG_NAME is set
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/origin-node
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -f /etc/sysconfig/atomic-openshift-node ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/atomic-openshift-node
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: elif [[ -f /etc/sysconfig/origin-node ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: file=/etc/sysconfig/origin-node
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Waiting for the node sysconfig file to be created" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 & wait
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: name="$(sed -nE 's|^BOOTSTRAP_CONFIG_NAME=([^#].+)|\1|p' "${file}" | head -1)"
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -z "${name}" ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Waiting for BOOTSTRAP_CONFIG_NAME to be set" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 & wait
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # in the background check to see if the value changes and exit if so
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: pid=$BASHPID
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: (
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! updated="$(sed -nE 's|^BOOTSTRAP_CONFIG_NAME=([^#].+)|\1|p' "${file}" | head -1)"; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to check for bootstrap config, exiting" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: kill $pid
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: exit 1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ "${updated}" != "${name}" ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Bootstrap configuration profile name changed, exiting" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: kill $pid
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: exit 0
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: ) &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: break
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: mkdir -p /etc/origin/node/tmp
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # periodically refresh both node-config.yaml and relabel the node
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: while true; do
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! oc extract "configmaps/${name}" -n openshift-node --to=/etc/origin/node/tmp --confirm --request-timeout=10s --config /etc/origin/node/node.kubeconfig "--token=$( cat /var/run/secrets/kubernetes.io/serviceaccount/token )"  > /dev/null; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to retrieve latest config for node" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 15 &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $!
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: KUBELET_HOSTNAME_OVERRIDE=$(cat /etc/sysconfig/KUBELET_HOSTNAME_OVERRIDE) || :
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! [[ -z "$KUBELET_HOSTNAME_OVERRIDE" ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: #Patching node-config for hostname override
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "nodeName: $KUBELET_HOSTNAME_OVERRIDE" >> /etc/origin/node/tmp/node-config.yaml
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # detect whether the node-config.yaml has changed, and if so trigger a restart of the kubelet.
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: md5sum /etc/origin/node/tmp/node-config.yaml > /tmp/.new
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ "$( cat /tmp/.old )" != "$( cat /tmp/.new )" ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: mv /etc/origin/node/tmp/node-config.yaml /etc/origin/node/node-config.yaml
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: SYSTEMD_IGNORE_CHROOT=1 systemctl restart tuned || :
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Configuration changed, restarting kubelet" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # TODO: kubelet doesn't relabel nodes, best effort for now
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # https://github.com/kubernetes/kubernetes/issues/59314
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if args="$(openshift-node-config --config /etc/origin/node/node-config.yaml)"; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: labels=$(tr ' ' '\n' <<<$args | sed -ne '/^--node-labels=/ { s/^--node-labels=//; p; }' | tr ',\n' ' ')
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if [[ -n "${labels}" ]]; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "info: Applying node labels $labels" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! oc label --config=/etc/origin/node/node.kubeconfig "node/${NODE_NAME}" ${labels} --overwrite; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to apply labels, will retry in 10" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $!
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: else
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: The downloaded node configuration is invalid, retrying later" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $!
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: if ! pkill -U 0 -f '(^|/)hyperkube kubelet '; then
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: echo "error: Unable to restart Kubelet" 2>&1
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 10 &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $!
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: continue
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: fi
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: # annotate node with md5sum of the config
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: oc annotate --config=/etc/origin/node/node.kubeconfig "node/${NODE_NAME}" \
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: node.openshift.io/md5sum="$( cat /tmp/.new | cut -d' ' -f1 )" --overwrite
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: cp -f /tmp/.new /tmp/.old
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: sleep 180 &
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: wait $!
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: done
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: ] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:host-config ReadOnly:false MountPath:/etc/origin/node/ SubPath: MountPropagation:<nil>} {Name:host-sysconfig-node ReadOnly:true MountPath:/etc/sysconfig SubPath: MountPropagation:<nil>} {Name:var-run-dbus ReadOnly:true MountPath:/var/run/dbus SubPath: MountPropagation:<nil>} {Name:run-systemd-system ReadOnly:true MountPath:/run/systemd/system SubPath: MountPropagation:<nil>} {Name:sync-token-g75k9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:&SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:*0,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,} Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.714272     504 kuberuntime_manager.go:757] checking backoff for container "sync" in pod "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)"
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.750145     504 cloud_request_manager.go:89] Requesting node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal"
abr 21 15:49:53 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:53.753161     504 cloud_request_manager.go:108] Node addresses from cloud provider for node "ip-10-x-x-37.ec2.internal" collected
abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem
abr 21 15:49:53 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Unmounting Filesystem
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: devmap not registered, can't remove
abr 21 15:49:54 ip-10-x-x-37.ec2.internal multipathd[1652]: dm-15: remove map (uevent)
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Mounting V5 Filesystem
abr 21 15:49:54 ip-10-x-x-37.ec2.internal kernel: XFS (dm-15): Ending clean mount
abr 21 15:49:54 ip-10-x-x-37.ec2.internal systemd[1]: Started libcontainer container f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c.
-- Subject: Unit docker-f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker-f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c.scope has finished starting up.
-- 
-- The start-up result is done.
abr 21 15:49:54 ip-10-x-x-37.ec2.internal oci-systemd-hook[80422]: systemdhook <debug>: f9365a3ea4b0: Skipping as container command is /bin/bash, not init or systemd
abr 21 15:49:54 ip-10-x-x-37.ec2.internal oci-umount[80423]: umounthook <debug>: prestart container_id:f9365a3ea4b0 rootfs:/var/lib/docker/devicemapper/mnt/9b4b9d6d74390bb00384017d66f53cf0d47637a72bc9d527bab7d26e395ade72/rootfs
abr 21 15:49:54 ip-10-x-x-37.ec2.internal atomic-openshift-node[504]: I0421 15:49:54.635000     504 kubelet.go:1865] SyncLoop (PLEG): "sync-qwxqn_openshift-node(b0d98acc-f9da-11e8-9d54-0ab51f65d96e)", event: &pleg.PodLifecycleEvent{ID:"b0d98acc-f9da-11e8-9d54-0ab51f65d96e", Type:"ContainerStarted", Data:"f9365a3ea4b09d41e74a09b242c63129fae6e1e37e1a6564016eb2cf0d547d0c"}
abr 21 15:49:58 ip-10-x-x-37.ec2.internal systemd[1]: Stopping Dynamic System Tuning Daemon...
-- Subject: Unit tuned.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit tuned.service has begun shutting down.
abr 21 15:49:59 ip-10-x-x-37.ec2.internal kernel: nf_conntrack: falling back to vmalloc.
abr 21 15:49:59 ip-10-x-x-37.ec2.internal systemd[1]: Stopped Dynamic System Tuning Daemon.
-- Subject: Unit tuned.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit tuned.service has finished shutting down.
abr 21 15:49:59 ip-10-x-x-37.ec2.internal systemd[1]: Starting Dynamic System Tuning Daemon...
-- Subject: Unit tuned.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit tuned.service has begun starting up.
abr 21 15:50:00 ip-10-x-x-37.ec2.internal kernel: nf_conntrack: falling back to vmalloc.
abr 21 15:50:00 ip-10-x-x-37.ec2.internal systemd[1]: Started Dynamic System Tuning Daemon.
-- Subject: Unit tuned.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit tuned.service has finished starting up.
-- 
-- The start-up result is done.
abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: atomic-openshift-node.service holdoff time over, scheduling restart.
abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: Stopped OpenShift Node.
-- Subject: Unit atomic-openshift-node.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit atomic-openshift-node.service has finished shutting down.
abr 21 15:50:05 ip-10-x-x-37.ec2.internal systemd[1]: Starting OpenShift Node...
-- Subject: Unit atomic-openshift-node.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit atomic-openshift-node.service has begun starting up.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --allow-privileged has been deprecated, will be removed in a future version
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authentication-token-webhook-cache-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-webhook-cache-authorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --authorization-webhook-cache-unauthorized-ttl has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cadvisor-port has been deprecated, The default will change to 0 (disabled) in 1.11, and the cadvisor port will be removed entirely in 1.12
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
abr 21 15:50:06 ip-10-x-x-37.ec2.internal atomic-openshift-node[425]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

Comment 1 rvanderp 2019-07-22 17:52:37 UTC
If there are any suggestions of things to look at or additional diagnostic information I can gather, I would really appreciate it.


Note You need to log in before you can comment on or make changes to this bug.