Bug 1508040

Description of problem:

The kubelet did not initialize modules properly, and it appeared the container manager failed to initialize or detect that the kubepods cgroup existed.  Manual inspection of the cgroupfs showed the cgroup existed, but the kubelet never recovered.  The pleg kept logging errors as follows as a result.

22496 kubelet.go:1788] skipping pod synchronization - [Failed to start ContainerManager failed to initialize top level QOS containers: root container /kubepods doesn't exist]

We need to check how we recover from errors when the kubelet fails to initialize modules.

oc version
oc v3.7.0-0.143.7
kubernetes v1.7.0+80709908fd
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://internal.api.starter-ca-central-1.openshift.com:443
openshift v3.7.0-0.143.7
kubernetes v1.7.0+80709908fd

node: ip-172-31-20-46.ca-central-1.compute.internal

 oc describe nodes ip-172-31-20-46.ca-central-1.compute.internal
Name:			ip-172-31-20-46.ca-central-1.compute.internal
Role:			
Labels:			beta.kubernetes.io/arch=amd64
			beta.kubernetes.io/instance-type=m4.2xlarge
			beta.kubernetes.io/os=linux
			failure-domain.beta.kubernetes.io/region=ca-central-1
			failure-domain.beta.kubernetes.io/zone=ca-central-1a
			hostname=starter-ca-central-1-node-compute-178d8
			kubernetes.io/hostname=ip-172-31-20-46.ca-central-1.compute.internal
			logging-infra-fluentd=false
			ops_node=old
			region=ca-central-1
			type=compute
Annotations:		volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:			<none>
CreationTimestamp:	Sat, 19 Aug 2017 02:42:07 +0000
Conditions:
  Type			Status	LastHeartbeatTime			LastTransitionTime			Reason				Message
  ----			------	-----------------			------------------			------				-------
  DiskPressure 		False 	Tue, 31 Oct 2017 17:38:16 +0000 	Mon, 30 Oct 2017 21:33:44 +0000 	KubeletHasNoDiskPressure 	kubelet has no disk pressure
  MemoryPressure 	False 	Tue, 31 Oct 2017 17:38:16 +0000 	Mon, 30 Oct 2017 21:33:44 +0000 	KubeletHasSufficientMemory 	kubelet has sufficient memory available
  OutOfDisk 		False 	Tue, 31 Oct 2017 17:38:16 +0000 	Mon, 30 Oct 2017 21:33:44 +0000 	KubeletHasSufficientDisk 	kubelet has sufficient disk space available
  Ready 		False 	Tue, 31 Oct 2017 17:38:16 +0000 	Mon, 30 Oct 2017 21:33:44 +0000 	KubeletNotReady 		Failed to start ContainerManager failed to initialize top level QOS containers: root container /kubepods doesn't exist
Addresses:
  InternalIP:	172.31.20.46
  ExternalIP:	35.182.74.104
  InternalDNS:	ip-172-31-20-46.ca-central-1.compute.internal
  ExternalDNS:	ec2-35-182-74-104.ca-central-1.compute.amazonaws.com
  Hostname:	ip-172-31-20-46.ca-central-1.compute.internal
Capacity:
 cpu:		8
 memory:	32780632Ki
 pods:		160
Allocatable:
 cpu:		7
 memory:	31678232Ki
 pods:		160
System Info:
 Machine ID:			d52c597d0f1a42aeb01b5a7d71e63f24
 System UUID:			EC21415F-3777-B253-CFD3-BFFD5A4C625A
 Boot ID:			e82d0b88-e066-4ea6-88e3-9c381c7463b7
 Kernel Version:		3.10.0-753.el7xtsync.x86_64
 OS Image:			Employee SKU
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://1.12.6
 Kubelet Version:		v1.7.0+80709908fd
 Kube-Proxy Version:		v1.7.0+80709908fd
ExternalID:			i-099506d6ac0b0046b
Non-terminated Pods:		(17 in total)
  Namespace			Name						CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ---------			----						------------	----------	---------------	-------------
  2-loro			loro-4-v85vn					120m (1%)	2 (28%)		614Mi (1%)	1Gi (3%)
  apolo				tomcat-3-q13kg					60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  arrr-matey			browser-2-v6r46					60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  brazarestaurant		brazarestaurant-2-4t6s4				60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  jamesludunbar			jamesludunbar-2-l2f0l				60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  jamilah			jamilah-1-kdv7z					105m (1%)	1757m (25%)	540Mi (1%)	900Mi (2%)
  javaee7-petclinic		javaee7-petclinic-3-cwll6			60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  javapg			openjdk-app-2-gdmkf				60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  lfhbeue			redhat-openjdk18-openshi-1-rhv03		60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  logging			logging-fluentd-smsvp				100m (1%)	0 (0%)		1536Mi (4%)	1536Mi (4%)
  manga-app			laravel-mysql-persistent-2-hook-pre		60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  mjclighting			mjclighting-2-mm8lj				60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  onlineshop2			mysql-1-tszjv					60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  routeplanner2			guide-1-241fs					60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  ugawgsu			redhat-openjdk18-openshi-1-81c4l		60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  willfood			test-111-nsr67					60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
  zuxrxddhbh			redhat-openjdk18-openshi-1-7g896		60m (0%)	1 (14%)		307Mi (0%)	512Mi (1%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  1165m (16%)	17757m (253%)	6988Mi (22%)	10628Mi (34%)


I restarted the the node process, and the agent recovered as expected.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: