On restart and at other times kubelet logs are full of messages like for both static pods and for restart Never regular pods.  We need to know why this is happening.

May 02 19:22:00 ip-10-0-133-133 hyperkube[2047]: E0502 19:22:00.049312    2047 kubelet_pods.go:1351] Pod attempted illegal phase transition from Succeeded to Failed: &PodStatus{Phase:Failed,Conditions:[],Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[{installer {nil nil ContainerStateTerminated{ExitCode:255,Signal:0,Reason:Error,Message:c/kubernetes/static-pod-resources/kube-scheduler-pod-8"}}],"initContainers":[{"name":"wait-for-host-port","image":"registry.svc.ci.openshift.org/ocp/4.1-2019-05-02-141058@sha256:81fd0b5628b5b7a8376546a63450512e1541cbaf6ddaa1df92bec7fc29b7fb9a","command":["/usr/bin/timeout","30","/bin/bash","-c"],"args":["echo -n \"Waiting for port :10259 and :10251 to be released.\"\nwhile [ -n \"$(lsof -ni :10251)\" -o -n \"$(lsof -i :10259)\" ]; do\n  echo -n \".\"\n  sleep 1\ndone\n"],"resources":{},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"}],"containers":[{"name":"scheduler","image":"registry.svc.ci.openshift.org/ocp/4.1-2019-05-02-141058@sha256:81fd0b5628b5b7a8376546a63450512e1541cbaf6ddaa1df92bec7fc29b7fb9a","command":["hyperkube","kube-scheduler"],"args":["--config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml","--cert-dir=/var/run/kubernetes","--port=0","--authentication-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig","--authorization-kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/scheduler-kubeconfig/kubeconfig","--feature-gates=ExperimentalCriticalPodAnnotation=true,LocalStorageCapacityIsolation=false,RotateKubeletServerCertificate=true,SupportPodPidsLimit=true","-v=2","--tls-cert-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.crt","--tls-private-key-file=/etc/kubernetes/static-pod-resources/secrets/serving-cert/tls.key"],"ports":[{"containerPort":10259}],"resources":{"requests":{"memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"}],"livenessProbe":{"httpGet":{"path":"healthz","port":10251,"scheme":"HTTP"},"initialDelaySeconds":45},"readinessProbe":{"httpGet":{"path":"healthz","port":10251,"scheme":"HTTP"},"initialDelaySeconds":45},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"}],"hostNetwork":true,"tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"},"status":{}}
May 02 19:22:00 ip-10-0-133-133 hyperkube[2047]: ,StartedAt:2019-05-02 18:49:05.095807071 +0000 UTC,FinishedAt:2019-05-02 19:21:50.8862873 +0000 UTC,ContainerID:cri-o://717ae8b1aa95b6025f3f095a76314a6aadbb17f97e87c7c330de016b375203b6,}} {nil nil nil} false 0 registry.svc.ci.openshift.org/ocp/4.1-2019-05-02-170923@sha256:b8d45a3efc4c8b5cf6c49ca621199b9bc7514e3ef473e76d2e16dd99bdc2ca33 registry.svc.ci.openshift.org/ocp/4.1-2019-05-02-170923@sha256:b8d45a3efc4c8b5cf6c49ca621199b9bc7514e3ef473e76d2e16dd99bdc2ca33 cri-o://717ae8b1aa95b6025f3f095a76314a6aadbb17f97e87c7c330de016b375203b6}],QOSClass:Burstable,InitContainerStatuses:[],NominatedNodeName:,}