Bug 1860163

Created attachment 1702270 [details]
image_count_graph

Description of problem:

On our 3.11 cluster used for CI we periodically prune images via a pod that looks like this:
```
    containers:
    - args:
      - adm
      - prune
      - images
      - --certificate-authority=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
      - --keep-tag-revisions=3
      - --keep-younger-than=96h
      - --confirm=true
      command:
      - oc
      image: quay.io/openshift/origin-cli:4.5
      imagePullPolicy: Always
      name: ""
      resources:
        limits:
          cpu: "2"
          memory: 8Gi
        requests:
          cpu: "1"
          memory: 4Gi
    serviceAccountName: image-pruner
```

The job always times out after four hours: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directory/rehearse-10177-periodic-ci-image-pruner

Impact:
This caused an outage of about 40 minutes today because the etcd went out of storage space. We worked around it by increasing the storage of etcd to 6gib from the default 4gib. This will only temporarily solve the problem as the number of images increases and once we are at 8gig we can not increase etcd storage further. This needs a fix until we hit that 8 gig limit or Openshift will not have CI anymore.


Attached a graph of the rising image count.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: