Bug 1860163 - Prunning of images doesn't appaer to be working
Summary: Prunning of images doesn't appaer to be working
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.6.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
: 1885545 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2020-07-23 19:53 UTC by aaleman
Modified: 2023-12-15 18:34 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the method for tracking dependencies between objects is slow Consequence: sometimes it was working more than 5 days without any visible progress Fix: change the underlying mechanism Result: the pruner has better parallelism and works faster
Clone Of:
Last Closed: 2020-10-27 16:16:56 UTC
Target Upstream Version:

Attachments (Terms of Use)
image_count_graph (46.04 KB, image/png)
2020-07-23 19:53 UTC, aaleman
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift oc pull 510 0 None closed Bug 1860163: Optimized Image Pruner 2021-02-18 03:02:17 UTC
Red Hat Knowledge Base (Solution) 4186901 0 None None None 2020-10-06 11:16:17 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:17:14 UTC

Description aaleman 2020-07-23 19:53:24 UTC
Created attachment 1702270 [details]

Description of problem:

On our 3.11 cluster used for CI we periodically prune images via a pod that looks like this:
    - args:
      - adm
      - prune
      - images
      - --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
      - --keep-tag-revisions=3
      - --keep-younger-than=96h
      - --confirm=true
      - oc
      image: quay.io/openshift/origin-cli:4.5
      imagePullPolicy: Always
      name: ""
          cpu: "2"
          memory: 8Gi
          cpu: "1"
          memory: 4Gi
    serviceAccountName: image-pruner

The job always times out after four hours: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directory/rehearse-10177-periodic-ci-image-pruner

This caused an outage of about 40 minutes today because the etcd went out of storage space. We worked around it by increasing the storage of etcd to 6gib from the default 4gib. This will only temporarily solve the problem as the number of images increases and once we are at 8gig we can not increase etcd storage further. This needs a fix until we hit that 8 gig limit or Openshift will not have CI anymore.

Attached a graph of the rising image count.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Steve Kuznetsov 2020-07-23 21:14:43 UTC
The job successfully ran for five days, but made no progress, either.

We MUST make changes to `oc adm prune` to make it run in parallel as needed or debugging this will be impossible.

Comment 6 Wenjing Zheng 2020-09-30 06:34:41 UTC
No bug found during regression test on image pruner, so verify this bug now on 
$ oc version
Client Version: 4.6.0-0.nightly-2020-09-29-170625
Server Version: 4.6.0-0.nightly-2020-09-29-170625
Kubernetes Version: v1.19.0+6ef2098

Comment 7 Oleg Bulatov 2020-10-06 11:16:18 UTC
*** Bug 1885545 has been marked as a duplicate of this bug. ***

Comment 8 Fatima 2020-10-06 11:24:12 UTC
*** Bug 1885545 has been marked as a duplicate of this bug. ***

Comment 14 errata-xmlrpc 2020-10-27 16:16:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.