Description of problem: When running a large environment, image data in the ectd database is contributing to large databases. This results hosts running etcd needing large amounts of System Memory. Version-Release number of selected component (if applicable): 3.2.x Additional info: After pruning images still contribute for more than 50% of the DB size. Some environment running with 1000+ project can have many images that lead to etcd being larger than is recommended by coreOS. DBs over 700mb and some larger that 2GB.
The PR that should move the manifests for new images back to registry is here: https://github.com/openshift/origin/pull/11925 This should stop growing the etcd by not storing the new manifests there. As the users will push new images, the old images and their manifests will be pruned from etcd and the storage will decrease. As a follow up we will probably have a tool to vacuum the image storage.
> in v3.3 we support v2 manifests which should reduce that load. Just a remark, that internal registry doesn't accept schema 2 by default - it's rejected which forces client to re-upload schema 1. It can be enabled though as noted in documentation. Also the switch to schema 2 won't help to reduce etcd size. The manifest of schema 2 is indeed smaller but there's also a manifest config which used to be embedded in manifest schema 1. We store it in etcd as well (just for the schema 2). The PR in Comment 5 will help a lot to reduce the size. Do we want to back-port it to 3.2, 3.3 or just 3.4?
https://github.com/openshift/origin/pull/11925 is related to this.
Still waiting for fork_amis ready for test on 3.3, thanks
Paul, we have co-work with miminar done some tests for 3.3, and it still need miminar provide some msg, pls reference to the background in bug: https://bugzilla.redhat.com/show_bug.cgi?id=1418358
@pweil I thinks it's sufficient to treat this as a 3.3/3.4 bug. Closing this in favor of bug 1418358. *** This bug has been marked as a duplicate of bug 1418358 ***