Bug 1827075 - [gcp] Image registry cannot be fully removed with 409 error about the deleting bucket is not empty
Summary: [gcp] Image registry cannot be fully removed with 409 error about the deletin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Adam Kaplan
QA Contact: Wenjing Zheng
URL:
Whiteboard:
: 1828008 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-23 08:27 UTC by Wenjing Zheng
Modified: 2020-07-13 17:30 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: image registry did not remove objects on GCS prior to removing the bucket Consequence: image registry could not be removed on GCP if images were pushed to the registry Fix: image registry operator removes all objects on GCS buckets prior to deletion Result: image registry can be removed on GCP
Clone Of:
Environment:
Last Closed: 2020-07-13 17:30:33 UTC
Target Upstream Version:


Attachments (Terms of Use)
Image registry operator log (1.93 MB, text/plain)
2020-04-23 08:37 UTC, Wenjing Zheng
no flags Details


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 554 None closed Bug 1827075: Drain GCS buckets before removal 2020-07-02 19:26:04 UTC
Red Hat Product Errata RHBA-2020:2409 None None None 2020-07-13 17:30:44 UTC

Description Wenjing Zheng 2020-04-23 08:27:40 UTC
Description of problem:
Below message appreas when try to remove non-empty image registry:
message: ’googleapi: Error 409: The bucket you tried to delete was not empty.,

Version-Release number of selected component (if applicable):
4.3.14

How reproducible:
Not always

Steps to Reproduce:
1.Import some images to image registry with GSC storage backend
2.Try to remove image registry
3.Try to bring image registry back and check pods under openshift-image-registry
$ oc get pods
NAME                                               READY   STATUS        RESTARTS   AGE
cluster-image-registry-operator-78c9b64968-dr6f5   2/2     Running       0          20h
image-registry-549fbdc64f-jlzmt                    0/1     Pending       0          81s
image-registry-69bd779cdd-9pbwg                    0/1     Terminating   0          82s
image-registry-7bb54df499-lpb8g                    0/1     Pending       0          81s
node-ca-79vw9                                      1/1     Running       1          20h
node-ca-chnkb                                      1/1     Running       1          20h
node-ca-w6csh                                      1/1     Running       1          20h
node-ca-x5kbv                                      1/1     Running       1          20h
node-ca-z2hzd                                      1/1     Running       1          20h


Actual results:
Image registry pod cannot come back.

Expected results:
It should be back.

Additional info:
Workaround: manually delete bucket from gcp console(operator will prompt "unable to sync: unable to finalize resource: unable to remove storage: googleapi: Error 404: Not Found, notFound, googleapi: Error 404: Not Found"), the bucket can be deleted in GCP console, then recreate a bucket with same name of the deleted one.

Comment 1 Wenjing Zheng 2020-04-23 08:37:35 UTC
Created attachment 1681032 [details]
Image registry operator log

Comment 3 Oleg Bulatov 2020-04-27 09:50:17 UTC
*** Bug 1828008 has been marked as a duplicate of this bug. ***

Comment 12 Wenjing Zheng 2020-05-26 11:20:32 UTC
Verified on 4.5.0-0.nightly-2020-05-26-063751.

Comment 13 errata-xmlrpc 2020-07-13 17:30:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.