Bug 1827075

Summary: [gcp] Image registry cannot be fully removed with 409 error about the deleting bucket is not empty
Product: OpenShift Container Platform Reporter: Wenjing Zheng <wzheng>
Component: Image RegistryAssignee: Adam Kaplan <adam.kaplan>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.zCC: adam.kaplan, aos-bugs
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: image registry did not remove objects on GCS prior to removing the bucket Consequence: image registry could not be removed on GCP if images were pushed to the registry Fix: image registry operator removes all objects on GCS buckets prior to deletion Result: image registry can be removed on GCP
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:30:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Image registry operator log none

Description Wenjing Zheng 2020-04-23 08:27:40 UTC 
Description of problem:
Below message appreas when try to remove non-empty image registry:
message: ’googleapi: Error 409: The bucket you tried to delete was not empty.,

Version-Release number of selected component (if applicable):
4.3.14

How reproducible:
Not always

Steps to Reproduce:
1.Import some images to image registry with GSC storage backend
2.Try to remove image registry
3.Try to bring image registry back and check pods under openshift-image-registry
$ oc get pods
NAME                                               READY   STATUS        RESTARTS   AGE
cluster-image-registry-operator-78c9b64968-dr6f5   2/2     Running       0          20h
image-registry-549fbdc64f-jlzmt                    0/1     Pending       0          81s
image-registry-69bd779cdd-9pbwg                    0/1     Terminating   0          82s
image-registry-7bb54df499-lpb8g                    0/1     Pending       0          81s
node-ca-79vw9                                      1/1     Running       1          20h
node-ca-chnkb                                      1/1     Running       1          20h
node-ca-w6csh                                      1/1     Running       1          20h
node-ca-x5kbv                                      1/1     Running       1          20h
node-ca-z2hzd                                      1/1     Running       1          20h


Actual results:
Image registry pod cannot come back.

Expected results:
It should be back.

Additional info:
Workaround: manually delete bucket from gcp console(operator will prompt "unable to sync: unable to finalize resource: unable to remove storage: googleapi: Error 404: Not Found, notFound, googleapi: Error 404: Not Found"), the bucket can be deleted in GCP console, then recreate a bucket with same name of the deleted one.
Comment 1 Wenjing Zheng 2020-04-23 08:37:35 UTC 
Created attachment 1681032 [details]
Image registry operator log
Comment 3 Oleg Bulatov 2020-04-27 09:50:17 UTC 
*** Bug 1828008 has been marked as a duplicate of this bug. ***
Comment 12 Wenjing Zheng 2020-05-26 11:20:32 UTC 
Verified on 4.5.0-0.nightly-2020-05-26-063751.
Comment 13 errata-xmlrpc 2020-07-13 17:30:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409