Bug 1887010 - Imagepruner met error "Job has reached the specified backoff limit" which causes image registry degraded
Summary: Imagepruner met error "Job has reached the specified backoff limit" which cau...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks: 1923993
TreeView+ depends on / blocked
 
Reported: 2020-10-10 04:03 UTC by Wenjing Zheng
Modified: 2021-02-24 15:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the pruner were trying to detect the registry name using image streams Consequence: when there were no image streams, the pruner failed to detect the registry name Fix: the image registry operator provides the pruner with the registry name Result: the pruner does not depend on existence of image streams
Clone Of:
Environment:
Last Closed: 2021-02-24 15:24:43 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 627 0 None closed Bug 1887010: Make pruner always aware of registry name 2021-02-17 18:21:28 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:25:21 UTC

Description Wenjing Zheng 2020-10-10 04:03:12 UTC 
Description of problem:
$ oc get co | grep image-registry
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
image-registry                             4.6.0-0.nightly-2020-10-09-033719   True        False         True       12h
$ oc get pods -n openshift-image-registry
NAME                                              READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-f8d5c5f8d-s9k8h   1/1     Running   1          12h
image-pruner-1602288000-t5454                     0/1     Error     0          148m
image-registry-799c867bbc-djb5m                   1/1     Running   0          12h
node-ca-chr7t                                     1/1     Running   0          12h
node-ca-fvzdc                                     1/1     Running   0          12h
node-ca-gmmhl                                     1/1     Running   0          12h
node-ca-h8h8j                                     1/1     Running   0          12h
node-ca-lnm6z                                     1/1     Running   0          12h
node-ca-pt9sw                                     1/1     Running   0          12h
$oc describe co image-registry
Status:
  Conditions:
    Last Transition Time:  2020-10-09T14:25:44Z
    Message:               Available: The registry is ready
ImagePrunerAvailable: Pruner CronJob has been created
    Reason:                Ready
    Status:                True
    Type:                  Available
    Last Transition Time:  2020-10-09T14:25:53Z
    Message:               Progressing: The registry is ready
    Reason:                Ready
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2020-10-10T00:00:06Z
    Message:               ImagePrunerDegraded: Job has reached the specified backoff limit
    Reason:                ImagePrunerJobFailed
    Status:                True
    Type:                  Degraded


Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-10-09-033719

How reproducible:
30%?

Steps to Reproduce:
1.Set up a cluster(for the cluster which can reproduce this issue is an openstack cluster)
2.
3.

Actual results:
Image registry is degraded for "ImagePrunerDegraded: Job has reached the specified backoff limit"

Expected results:
Should has no such error.

Additional info:
Workaround is to change some imagepruner config settings to bring it back.
Comment 5 Wenjing Zheng 2020-11-09 08:23:10 UTC 
Verified on 4.7.0-0.nightly-2020-11-08-225909:
    Args:
      adm
      prune
      images
      --confirm=true
      --certificate-authority=/var/run/configmaps/serviceca/service-ca.crt
      --keep-tag-revisions=3
      --keep-younger-than=60m
      --ignore-invalid-refs=true
      --loglevel=2
      --prune-registry=true
      --registry-url=https://image-registry.openshift-image-registry.svc:5000
Comment 14 errata-xmlrpc 2021-02-24 15:24:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633
Note You need to log in before you can comment on or make changes to this bug.