DescriptionAlexander Koksharov
2016-08-30 11:40:04 UTC
Description of problem:
Build start failing due to error updating imagestream. It looks like imageStream json object size reaches some maximum value and it is not possible to push it to etcd any more.
We have retrieved the image-stream sizes on the two prod environments, where the build are not working anymore due to the given issue..
environment1:
oc get is -o json | grep dockerImageReference | wc -l
-> 3187
oc get is -o json | wc --bytes
-> 1467191
environment2:
oc get is -o json | grep dockerImageReference | wc -l
-> 3188
oc get is -o json | wc --bytes
-> 1467595
Version-Release number of selected component (if applicable):
3.2
How reproducible:
automatically trigger build ('oc start-build test-app') periodically. It it will stop working after ~3188 builds.
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
there should be no limits
Additional info:
I would also suggest to do pruning occasionally when triggering 3188 builds.
Mid term, we will probably look into off-loading image manifests into an external storage, but that won't happen for 3.4.
(In reply to Michal Fojtik from comment #2)
> I would also suggest to do pruning occasionally when triggering 3188 builds.
>
> Mid term, we will probably look into off-loading image manifests into an
> external storage, but that won't happen for 3.4.
This is already being requested in https://bugzilla.redhat.com/show_bug.cgi?id=1378180
(In reply to Clayton Coleman from comment #1)
> Pruning should be run occasionally. We can also consider whether to allow
> users to opt-out of automatic old tag cleanup.
I think what you are referring to is https://trello.com/c/WsnlyxUf/805-use-scheduledjobs-to-prune-images
Description of problem: Build start failing due to error updating imagestream. It looks like imageStream json object size reaches some maximum value and it is not possible to push it to etcd any more. We have retrieved the image-stream sizes on the two prod environments, where the build are not working anymore due to the given issue.. environment1: oc get is -o json | grep dockerImageReference | wc -l -> 3187 oc get is -o json | wc --bytes -> 1467191 environment2: oc get is -o json | grep dockerImageReference | wc -l -> 3188 oc get is -o json | wc --bytes -> 1467595 Version-Release number of selected component (if applicable): 3.2 How reproducible: automatically trigger build ('oc start-build test-app') periodically. It it will stop working after ~3188 builds. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: there should be no limits Additional info: