Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1851414

Summary:	`oc adm prune images` should retry on conflicts
Product:	OpenShift Container Platform	Reporter:	Bruno Barcarol Guimarães <bbarcaro>
Component:	Image Registry	Assignee:	Oleg Bulatov <obulatov>
Status:	CLOSED NOTABUG	QA Contact:	Wenjing Zheng <wzheng>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	aos-bugs, wking
Target Milestone:	---	Flags:	bbarcaro: needinfo-
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-08-17 14:21:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Bruno Barcarol Guimarães 2020-06-26 13:16:46 UTC

Description of problem:

We have a periodic CI job that executes `oc adm prune images` daily:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=periodic-ci-image-pruner

It frequently fails with:

error updating image stream $x…: the object has been modified; please apply your changes to the latest version and try again

How reproducible:

Almost every execution fails with that error, likely due to the volume of IS operations in the CI cluster:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/job-history/gs/origin-ci-test/logs/periodic-ci-image-pruner

Comment 1 Oleg Bulatov 2020-06-26 14:35:11 UTC

Setting the version to 4.5 as the job uses `quay.io/openshift/origin-cli:4.5`.

Comment 2 Oleg Bulatov 2020-07-09 21:19:44 UTC

That's not the reason why this job fails, there are retries in the pruner. The real reason is OOM killer:

https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/origin-ci-test/logs/periodic-ci-image-pruner/1275217546462629888

    state:
      terminated:
        containerID: docker://2ded865fc54bac1e7435aeb2c98122028fefae87c37c883a78a0ae91df252022
        exitCode: 255
        finishedAt: "2020-06-23T00:11:04Z"
        ...
        reason: OOMKilled

Can you increase memory limits?