Bug 1538215

Summary: [DOCKER] Eviction manager erros in node logs
Product: OpenShift Container Platform Reporter: Vikas Laad <vlaad>
Component: ContainersAssignee: Antonio Murdaca <amurdaca>
Status: CLOSED CURRENTRELEASE QA Contact: Vikas Laad <vlaad>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: amurdaca, aos-bugs, jokerman, lsm5, mitr, mmccomas, smunilla
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift v3.9.19 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-18 18:17:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vikas Laad 2018-01-24 16:04:33 UTC 
Description of problem:
Lots of Eviction Manager errors in node logs as follows

Jan 20 23:11:49 ip-172-31-13-204.us-west-2.compute.internal atomic-openshift-node[5748]: E0120 23:11:49.959266    5748 eviction_manager.go:238] eviction manager: unexpected err: failed to get imageFs stats: failed to get image stats: rpc error: code = Unknown desc = Error response from daemon: layer does not exist

Version-Release number of selected component (if applicable):
openshift v3.9.0-0.20.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8

Steps to Reproduce:
1. Keep creating/building/scaling quickstart apps on OCP cluster
2. See node logs for errors

Actual results:
Eviction errors in node logs

Expected results:
No errors in node logs

Additional info:
Attaching node logs
Comment 7 Antonio Murdaca 2018-02-06 16:29:11 UTC 
fixed in latest branches for 1.12/1.13
Comment 10 Vikas Laad 2018-04-16 13:43:24 UTC 
Verified in following version, did not see the problem after running reliability tests for a week.

openshift v3.9.19
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Tue Feb 20 13:46:34 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: <unknown>
 Go version:      go1.8.3
 Git commit:      774336d/1.13.1
 Built:           Tue Feb 20 13:46:34 2018
 OS/Arch:         linux/amd64
 Experimental:    false