Bug 1703521

Summary:	Deleted projects are stuck in "Terminating" state, even if there are no associated resources with them
Product:	OpenShift Container Platform	Reporter:	Udi Kalifon <ukalifon>
Component:	openshift-apiserver	Assignee:	Lukasz Szaszkiewicz <lszaszki>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Xingxing Xia <xxia>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.1.0	CC:	aos-bugs, eparis, gblomqui, jokerman, mfojtik, mmccomas, sttts
Target Milestone:	---
Target Release:	4.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-05 09:06:46 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Udi Kalifon 2019-04-26 15:56:34 UTC

Description of problem:
When simply creating and deleting a project, even without creating any resources in that project, it still doesn't get fully deleted.

There is a knowledge base article about it: https://access.redhat.com/solutions/3263161

However, the article assumes that the reason for the issue is that the user didn't first delete all associated resources. It seems that the bug simply happens ever time now.

I verified that there are no associated resources by running:
oc get all -n <project>


Version-Release number of selected component (if applicable):
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0", GitCommit:"04203bcb0", GitTreeState:"clean", BuildDate:"2019-04-26T10:41:14Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+938b976", GitCommit:"938b976", GitTreeState:"clean", BuildDate:"2019-04-17T12:43:17Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}


How reproducible:
100%


Steps to Reproduce:
1. oc new-project blah
2. oc delete project blah
3. Open the GUI - you will see that the project never really gets completely deleted


Actual results:
Project is stuck in "Terminating" state


Expected results:
Project gone


Additional info:
The installation is on libvirt, using the dev-scripts which are used by the kubevirt and metalkube teams.

Comment 1 Eric Paris 2019-04-26 16:43:14 UTC

can you send `oc get -o yaml namespace $NAMESPACE`

Comment 2 Udi Kalifon 2019-04-27 20:08:02 UTC

oc get -o yaml namespace blah
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/description: ""
    openshift.io/display-name: ""
    openshift.io/requester: kube:admin
    openshift.io/sa.scc.mcs: s0:c24,c4
    openshift.io/sa.scc.supplemental-groups: 1000560000/10000
    openshift.io/sa.scc.uid-range: 1000560000/10000
  creationTimestamp: "2019-04-26T13:39:55Z"
  deletionTimestamp: "2019-04-26T13:40:53Z"
  name: blah
  resourceVersion: "1438747"
  selfLink: /api/v1/namespaces/blah
  uid: c6ad3776-6828-11e9-98d6-525400cdbd46
spec:
  finalizers:
  - kubernetes
status:
  phase: Terminating

Comment 7 Lukasz Szaszkiewicz 2019-08-16 10:27:28 UTC

I'm unable to reproduce the issue with the following version:

oc version                   
Client Version: version.Info{Major:"4", Minor:"2+", GitVersion:"v4.2.0-201908071853+7b9a22b-dirty", GitCommit:"7b9a22b", GitTreeState:"dirty", BuildDate:"2019-08-07T23:34:07Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0+ea1bbc7", GitCommit:"ea1bbc7", GitTreeState:"clean", BuildDate:"2019-08-16T00:00:58Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
OpenShift Version: 4.2.0-0.ci-2019-08-16-072837

I did:

1. oc new-project blah
Already on project "blah" on server "https://api.ci-ln-0ygjbd2-d5d6b.origin-ci-int-aws.dev.rhcloud.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app django-psql-example

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node

2.oc delete project blah
project.project.openshift.io "blah" deleted

3.oc get -o yaml namespace blah
Error from server (NotFound): namespaces "blah" not found


I'm going to try with "4.1"

Comment 8 Lukasz Szaszkiewicz 2019-08-19 09:59:04 UTC

I'm unable to reproduce the issue with "4.1" version, I did:

1. oc version                                        
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0+89b3000-215-dirty", GitCommit:"89b3000", GitTreeState:"dirty", BuildDate:"2019-08-13T01:49:54Z", GoVersion:"go1.11.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+89b3000", GitCommit:"89b3000", GitTreeState:"clean", BuildDate:"2019-08-13T01:46:32Z", GoVersion:"go1.11.9", Compiler:"gc", Platform:"linux/amd64"}

2.oc new-project blah
Now using project "blah" on server "https://api.polynomial-po.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app django-psql-example

to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node

3.oc delete project blah                   
project.project.openshift.io "blah" deleted

4. oc get -o yaml namespace blah                      
Error from server (NotFound): namespaces "blah" not found

Comment 9 Lukasz Szaszkiewicz 2019-08-19 12:51:19 UTC

@Udi, could you increase the verbosity of kube controller manager (-v=5) and then run the steps you mentioned? I'd like to examine the logs from the namespace controller.

Comment 11 Greg Blomquist 2019-08-26 13:44:28 UTC

Udi, see comment #9

Comment 12 Udi Kalifon 2019-09-05 09:06:46 UTC

This is no longer an issue in current openshift versions. Closing.