Bug 1851414 - `oc adm prune images` should retry on conflicts
Summary: `oc adm prune images` should retry on conflicts
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.6.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-26 13:16 UTC by Bruno Barcarol Guimarães
Modified: 2020-11-02 09:03 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-17 14:21:50 UTC
Target Upstream Version:
Embargoed:
bbarcaro: needinfo-


Attachments (Terms of Use)

Description Bruno Barcarol Guimarães 2020-06-26 13:16:46 UTC
Description of problem:

We have a periodic CI job that executes `oc adm prune images` daily:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=periodic-ci-image-pruner

It frequently fails with:

error updating image stream $x…: the object has been modified; please apply your changes to the latest version and try again

How reproducible:

Almost every execution fails with that error, likely due to the volume of IS operations in the CI cluster:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/job-history/gs/origin-ci-test/logs/periodic-ci-image-pruner

Comment 1 Oleg Bulatov 2020-06-26 14:35:11 UTC
Setting the version to 4.5 as the job uses `quay.io/openshift/origin-cli:4.5`.

Comment 2 Oleg Bulatov 2020-07-09 21:19:44 UTC
That's not the reason why this job fails, there are retries in the pruner. The real reason is OOM killer:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/periodic-ci-image-pruner/1275217546462629888

    state:
      terminated:
        containerID: docker://2ded865fc54bac1e7435aeb2c98122028fefae87c37c883a78a0ae91df252022
        exitCode: 255
        finishedAt: "2020-06-23T00:11:04Z"
        ...
        reason: OOMKilled

Can you increase memory limits?


Note You need to log in before you can comment on or make changes to this bug.