Hide Forgot
Description of problem: Build start failing due to error updating imagestream. It looks like imageStream json object size reaches some maximum value and it is not possible to push it to etcd any more. We have retrieved the image-stream sizes on the two prod environments, where the build are not working anymore due to the given issue.. environment1: oc get is -o json | grep dockerImageReference | wc -l -> 3187 oc get is -o json | wc --bytes -> 1467191 environment2: oc get is -o json | grep dockerImageReference | wc -l -> 3188 oc get is -o json | wc --bytes -> 1467595 Version-Release number of selected component (if applicable): 3.2 How reproducible: automatically trigger build ('oc start-build test-app') periodically. It it will stop working after ~3188 builds. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: there should be no limits Additional info:
Pruning should be run occasionally. We can also consider whether to allow users to opt-out of automatic old tag cleanup.
I would also suggest to do pruning occasionally when triggering 3188 builds. Mid term, we will probably look into off-loading image manifests into an external storage, but that won't happen for 3.4.
(In reply to Michal Fojtik from comment #2) > I would also suggest to do pruning occasionally when triggering 3188 builds. > > Mid term, we will probably look into off-loading image manifests into an > external storage, but that won't happen for 3.4. This is already being requested in https://bugzilla.redhat.com/show_bug.cgi?id=1378180 (In reply to Clayton Coleman from comment #1) > Pruning should be run occasionally. We can also consider whether to allow > users to opt-out of automatic old tag cleanup. I think what you are referring to is https://trello.com/c/WsnlyxUf/805-use-scheduledjobs-to-prune-images
Added Trello tracker.
*** Bug 1661212 has been marked as a duplicate of this bug. ***