Bug 2051692 - co/image-registry is degrade because ImagePrunerDegraded: Job has reached the specified backoff limit
Summary: co/image-registry is degrade because ImagePrunerDegraded: Job has reached the...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.10.0
Assignee: Oleg Bulatov
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On: 1990125
Blocks: 2059606
TreeView+ depends on / blocked
 
Reported: 2022-02-07 19:16 UTC by OpenShift BugZilla Robot
Modified: 2022-03-10 16:43 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the image pruner job didn't retry on failures. Consequence: a single failure could make the image registry operator degraded until next run of the pruner. Fix: the job makes few more attempts if the pruner fails. Result: temporary problems with the pruner doesn't make the image registry operator degraded.
Clone Of:
Environment:
Last Closed: 2022-03-10 16:43:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 755 0 None open [release-4.10] Bug 2051692: Retry on pruner failures 2022-02-07 19:17:06 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:43:45 UTC

Comment 2 XiuJuan Wang 2022-02-08 13:14:07 UTC
Since the api timeout is hard to reproduce, I choose to set ignoreInvalidImageReferences:false to test on 4.10.0-0.nightly-2022-02-08-071248 cluster,

1.set 
      ignoreInvalidImageReferences:false
      schedule: '* * * * *'
2. Create a pod with invalid image name
3.Check the imagepruner pod, it will retry 5 times

 oc logs -f image-pruner-27405425-28886
I0208 13:05:03.810103       7 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made
attempt #1 has failed (exit code 1), going to make another attempt...
I0208 13:05:34.540589      16 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made
attempt #2 has failed (exit code 1), going to make another attempt...
I0208 13:06:35.232914      25 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made
attempt #3 has failed (exit code 1), going to make another attempt...
I0208 13:08:05.993120      34 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made
attempt #4 has failed (exit code 1), going to make another attempt...
I0208 13:10:06.650628      43 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made
attempt #5 has failed (exit code 1), going to make another attempt...
I0208 13:12:37.318843      52 prune.go:347] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Failed to build graph!

The following objects have invalid references:

  pod/prune1 namespace=wxj: container prune1: invalid image reference "quay.io/openshifttest/hello-pod@sha:123": invalid reference format

Either fix the references or delete the objects to make the pruner proceed.
error: failed to build graph - no changes made

Comment 5 errata-xmlrpc 2022-03-10 16:43:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.