Bug 1743273

Summary: Project stuck in Terminating
Product: OpenShift Container Platform Reporter: jnaess
Component: kube-controller-managerAssignee: Maciej Szulik <maszulik>
Status: CLOSED WONTFIX QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.1.zCC: aos-bugs, eparis, jokerman, mfojtik
Target Milestone: ---   
Target Release: 4.1.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1745567 (view as bug list) Environment:
Last Closed: 2020-01-20 18:16:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1745567    
Bug Blocks:    
Attachments:
Description Flags
log files from controller pods and etcd pods none

Description jnaess 2019-08-19 14:07:01 UTC 
Description of problem: Namespace locked in "Terminating" state.


Version-Release number of selected component (if applicable): 4.1.6


How reproducible: unknown


Steps to Reproduce:
1. oc delete project codeready
2. oc get projects

Actual results:
Error from server (Conflict): Operation cannot be fulfilled on namespaces "codeready": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Expected results:
Project deleted.

Additional info:
Looking online, there are some issues reported on github regarding this.
https://github.com/kubernetes/kubernetes/issues/60807

It seems to be an issue with "finalizers" in the spec of the project.
Deleting the line "- kubernetes" via oc edit project does not work. However, doing the following does fix it: https://success.docker.com/article/kubernetes-namespace-stuck-in-terminating

Raised to get awareness of it in OpenShift 4.1.z
Comment 1 Michal Fojtik 2019-08-26 09:55:35 UTC 
Can you follow https://docs.google.com/document/d/13YIo4z8r2U5h03cwJVADALHN_dYT0Odp7rgvnUo3TdE/edit?usp=sharing this document, especially the "For Garbage Collector or not able to delete API resources" ?
Comment 2 jnaess 2019-08-26 13:41:49 UTC 
Created attachment 1608130 [details]
log files from controller pods and etcd pods
Comment 3 jnaess 2019-08-26 13:43:28 UTC 
This is a rather new cluster. it was installed, has been idling for a coule of weeks, then blocked the upgrade that day due to the namespace terminating issue.
Nothing is really running on it apart from what is provided by openshift.
Comment 4 jnaess 2019-08-26 13:44:22 UTC 
Comment on attachment 1608130 [details]
log files from controller pods and etcd pods

cluster domain name has been replaced with example.com to anonymize the log files.