Bug 1392254

Summary: On scale up/down: Unable to inspect docker image while inspecting container - error or log spam
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED DUPLICATE QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, jokerman, mifiedle, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-08 18:35:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
node log none

Description Mike Fiedler 2016-11-07 00:29:08 UTC 
Description of problem:

Running tests which do hundreds of scale up/downs, the logs get spammed with the following E/Error level message on every scaleup:

Nov  6 19:20:56 ip-172-31-24-197 atomic-openshift-node: E1106 19:20:56.502088   13169 docker_manager.go:404] unable to inspect docker image "POD" while inspecting docker container "sha256:1924b34d94a8d8b77aa6e7742dbd17ea5c4587a9ef68b66b6d9bb70138322030": no such image: "sha256:1924b34d94a8d8b77aa6e7742dbd17ea5c4587a9ef68b66b6d9bb70138322030"

The scale up seems to work ok.  Reduce the loglevel on this message?  or investigate as a real Error?

This is with SELinux disabled and using overlayfs as the docker storage


Version-Release number of selected component (if applicable):   3.4.0.22


How reproducible: Always


Steps to Reproduce:
1.  Install 3.4.0.22 with Docker 1.12.3-3 using overlayfs.  Disable SELinux
2.  oc new-app --template=cakephp-mysql-example and wait for it to build
3.  oc scale --replicas=20 dc/cakephp-mysql-example
4.  Watch syslogs for E messages

Actual results:

24 instances of the above message and a successful scaleup.   Potential underlying issue?

Expected results:

Quiet logs for a successful scaleup.

Additional info:
Comment 1 Mike Fiedler 2016-11-07 00:30:46 UTC 
Created attachment 1217842 [details]
node log
Comment 2 Mike Fiedler 2016-11-07 00:40:49 UTC 
These 24 messages are also accompanied by 24 instances of this error message:

E1106 19:20:58.023842   13169 docker_manager.go:761] Logging security options: {key:seccomp value:unconfined msg:}
Comment 3 Mike Fiedler 2016-11-07 00:45:53 UTC 
On scale down an addition 28 instances of the message pop:

Nov  6 19:42:17 ip-172-31-24-197 atomic-openshift-node: E1106 19:42:17.533707   13169 docker_manager.go:404] unable to inspect docker image "cakephp-mysql-example" while inspecting docker container "sha256:1f1ffe3f6f0285686192496d15209831d74eb02959c9e4b6c6debe60b624121b": no such image: "sha256:1f1ffe3f6f0285686192496d15209831d74eb02959c9e4b6c6debe60b624121b"


Scale up 20 and scale down to 0 yields gives us 52 of the above errors.
Comment 4 Seth Jennings 2016-11-08 18:35:05 UTC 
This issue is addressed by the PRs (one still pending merge) in
https://bugzilla.redhat.com/show_bug.cgi?id=1389183

https://github.com/openshift/origin/pull/11674
https://github.com/openshift/origin/pull/11793

I was able to recreate on master but not on the branch for PR 11793.

*** This bug has been marked as a duplicate of bug 1389183 ***