Bug 1378180

Summary: Image data stored in ETCD accounting for large DB.
Product: OpenShift Container Platform Reporter: Ryan Howe <rhowe>
Component: Image RegistryAssignee: Michal Minar <miminar>
Status: CLOSED DUPLICATE QA Contact: ge liu <geliu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.2.1CC: aos-bugs, bleanhar, byount, erich, geliu, haowang, jeder, jmorales, jokerman, maszulik, mfojtik, miminar, mmccomas, pep, pweil, sdodson, trankin, tstclair
Target Milestone: ---Keywords: Performance
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1418358 1418359 (view as bug list) Environment:
Last Closed: 2017-03-07 13:08:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1267746, 1418358, 1418359    

Description Ryan Howe 2016-09-21 18:08:16 UTC
Description of problem:

  When running a large environment, image data in the ectd database is contributing to large databases. This results hosts running etcd needing large amounts of System Memory.  


Version-Release number of selected component (if applicable):
3.2.x

Additional info:

After pruning images still contribute for more than 50% of the DB size. Some environment running with 1000+ project can have many images that lead to etcd being larger than is recommended by coreOS. DBs over 700mb and some larger that 2GB.

Comment 5 Michal Fojtik 2016-11-24 10:32:35 UTC
The PR that should move the manifests for new images back to registry is here: https://github.com/openshift/origin/pull/11925

This should stop growing the etcd by not storing the new manifests there. As the users will push new images, the old images and their manifests will be pruned from etcd and the storage will decrease. As a follow up we will probably have a tool to vacuum the image storage.

Comment 6 Michal Minar 2016-12-06 14:43:10 UTC
> in v3.3 we support v2 manifests which should reduce that load.

Just a remark, that internal registry doesn't accept schema 2 by default - it's rejected which forces client to re-upload schema 1.

It can be enabled though as noted in documentation.

Also the switch to schema 2 won't help to reduce etcd size. The manifest of schema 2 is indeed smaller but there's also a manifest config which used to be embedded in manifest schema 1. We store it in etcd as well (just for the schema 2).

The PR in Comment 5 will help a lot to reduce the size. Do we want to back-port it to 3.2, 3.3 or just 3.4?

Comment 7 Eric Rich 2016-12-06 16:11:38 UTC
https://github.com/openshift/origin/pull/11925 is related to this.

Comment 34 ge liu 2017-03-06 09:51:15 UTC
Still waiting for fork_amis ready for test on 3.3, thanks

Comment 36 ge liu 2017-03-07 02:36:27 UTC
Paul, we have co-work with miminar done some tests for 3.3, and it still need miminar provide some msg, pls reference to the background in bug: https://bugzilla.redhat.com/show_bug.cgi?id=1418358

Comment 38 Michal Minar 2017-03-07 13:08:43 UTC
@pweil I thinks it's sufficient to treat this as a 3.3/3.4 bug. Closing this in favor of bug 1418358.

*** This bug has been marked as a duplicate of bug 1418358 ***