Bug 1372001

Summary: Unable to list or prune image
Product: OpenShift Container Platform Reporter: Miheer Salunke <misalunk>
Component: NodeAssignee: Derek Carr <decarr>
Status: CLOSED WONTFIX QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: high    
Version: 3.2.0CC: agoldste, aos-bugs, eparis, erich, jokerman, misalunk, mmccomas, ndordet, stwalter, tstclair
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-26 19:57:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
runtime stack none

Description Miheer Salunke 2016-08-31 16:22:37 UTC
Created attachment 1196458 [details]
runtime stack

Description of problem:

When doing a trying to list images, we've got an our of memory error :
oc get image
fatal error: runtime: out of memory

Providing the runtime stack in attached runtime.txt file

Version-Release number of selected component (if applicable):
3.2.0

How reproducible:
On customer end

Steps to Reproduce:
1.oc get images 
2.
3.

Actual results:
oc get images gives out of memory error

Expected results:
oc get images shall work fine

Additional info:
We are having a lot of memory consumption in production

Comment 4 Andy Goldstein 2016-08-31 20:44:52 UTC
Miheer,

When they tried to prune or list images, how much free memory did the system have where they were running the command? How much total memory does that system have?

Do you know how many images they have in etcd?

Do you know what the size of their etcd database is?

Thanks,
Andy

Comment 5 Andy Goldstein 2016-08-31 20:56:14 UTC
FYI if an attempt to run 'oc get images' or 'oadm prune images' runs out of memory, as we see in https://bugzilla.redhat.com/attachment.cgi?id=1196458, it is the system where the oc/oadm command is executed that is running out of memory and not necessarily the system where atomic-openshift-* is running.

Comment 12 Andy Goldstein 2016-09-02 18:04:39 UTC
Here is the code for a script that can display the etcd usage sorted by largest entries: https://github.com/ncdc/etcdstats

I'm also attaching a precompiled binary you can use

Comment 13 Andy Goldstein 2016-09-02 18:17:39 UTC
I can't attach the binary as it's larger than the limit. Will need to find an alternative.