Bug 1887010
| Summary: | Imagepruner met error "Job has reached the specified backoff limit" which causes image registry degraded | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenjing Zheng <wzheng> |
| Component: | Image Registry | Assignee: | Oleg Bulatov <obulatov> |
| Status: | CLOSED ERRATA | QA Contact: | Wenjing Zheng <wzheng> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.6 | CC: | aos-bugs, miabbott, scuppett |
| Target Milestone: | --- | Keywords: | Regression, UpcomingSprint |
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: the pruner were trying to detect the registry name using image streams
Consequence: when there were no image streams, the pruner failed to detect the registry name
Fix: the image registry operator provides the pruner with the registry name
Result: the pruner does not depend on existence of image streams
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 15:24:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1923993 | ||
Verified on 4.7.0-0.nightly-2020-11-08-225909:
Args:
adm
prune
images
--confirm=true
--certificate-authority=/var/run/configmaps/serviceca/service-ca.crt
--keep-tag-revisions=3
--keep-younger-than=60m
--ignore-invalid-refs=true
--loglevel=2
--prune-registry=true
--registry-url=https://image-registry.openshift-image-registry.svc:5000
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |
Description of problem: $ oc get co | grep image-registry NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE image-registry 4.6.0-0.nightly-2020-10-09-033719 True False True 12h $ oc get pods -n openshift-image-registry NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-f8d5c5f8d-s9k8h 1/1 Running 1 12h image-pruner-1602288000-t5454 0/1 Error 0 148m image-registry-799c867bbc-djb5m 1/1 Running 0 12h node-ca-chr7t 1/1 Running 0 12h node-ca-fvzdc 1/1 Running 0 12h node-ca-gmmhl 1/1 Running 0 12h node-ca-h8h8j 1/1 Running 0 12h node-ca-lnm6z 1/1 Running 0 12h node-ca-pt9sw 1/1 Running 0 12h $oc describe co image-registry Status: Conditions: Last Transition Time: 2020-10-09T14:25:44Z Message: Available: The registry is ready ImagePrunerAvailable: Pruner CronJob has been created Reason: Ready Status: True Type: Available Last Transition Time: 2020-10-09T14:25:53Z Message: Progressing: The registry is ready Reason: Ready Status: False Type: Progressing Last Transition Time: 2020-10-10T00:00:06Z Message: ImagePrunerDegraded: Job has reached the specified backoff limit Reason: ImagePrunerJobFailed Status: True Type: Degraded Version-Release number of selected component (if applicable): 4.6.0-0.nightly-2020-10-09-033719 How reproducible: 30%? Steps to Reproduce: 1.Set up a cluster(for the cluster which can reproduce this issue is an openstack cluster) 2. 3. Actual results: Image registry is degraded for "ImagePrunerDegraded: Job has reached the specified backoff limit" Expected results: Should has no such error. Additional info: Workaround is to change some imagepruner config settings to bring it back.