Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1540779

Summary: Project namespace stuck in terminating status
Product: OpenShift Container Platform Reporter: Kenjiro Nakayama <knakayam>
Component: openshift-apiserverAssignee: Stefan Schimanski <sttts>
Status: CLOSED EOL QA Contact: Wang Haoran <haowang>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: admin, andcosta, aos-bugs, daxuyin, dcaldwel, dmoessne, fgrosjea, fshaikh, gerodrig, gucore, haowang, hgomes, jjerezro, jokerman, knakayam, lfilho, maupadhy, mfojtik, mmccomas, mtaru, pprakash, rhowe, rjanik, romank
Target Milestone: ---Flags: haowang: needinfo-
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-07 11:21:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenjiro Nakayama 2018-02-01 00:54:19 UTC
Description of problem:

- namespace stuck in "Terminating" status
- We confirmed that when we try to delete same error is seen as https://github.com/openshift/origin/issues/18125

  # oc delete project test2 --force
  Error from server (Conflict): Operation cannot be fulfilled on namespaces "test2": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

  # oc delete project test2 --force --grace-period=0
  warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
  Error from server (Conflict): Operation cannot be fulfilled on namespaces "test2": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Version-Release number of selected component (if applicable):
- OCP 3.7

How reproducible: not 100%

Steps to Reproduce:
1. Not sure what is the trigger.

Actual results:

  # oc get projects -o wide | grep -i terminating
  test2                  test2                            Terminating
  test3                  test3                            Terminating

Expected results:
- namespace stuck in "Terminating" status

Additional info:
- As mentioned above, similar issue with https://github.com/openshift/origin/issues/18125

Comment 1 Kenjiro Nakayama 2018-02-01 05:17:09 UTC
Hi, as per upstream issue discussion, we checked resources remained in the project, but there are no such resources.

  # oc get all -n test2
  No resources found.

  # oc get project test2
  NAME      DISPLAY NAME   STATUS
  test2     test2          Terminating

Is there any workaround for this issue? If no workaround, we would like you to prioritize this ticket.

[1] https://github.com/openshift/origin/issues/18125

Comment 4 Dan Mace 2018-02-06 18:21:17 UTC
Kenjiro,

The /cases/02022112 directory seems to have disappeared from collab-shell.usersys.redhat.com.

I think the controller logs are the best diagnostic tool right now; could you try once more to provide the relevant logs? Thanks!

Comment 22 Richard Janík 2019-02-06 08:49:19 UTC
Hi,

we're running a 3.11 cluster and it looks like we hit something similar. We have an ansible playbook doing changes to an OCP cluster installed by an external tool, so the exact process may be hard to trace, if that's necessary. At one point, ansible attempts to run: oc delete project install-test , and this command has been stuck for 15+ hours. Logging in on the machine and investigating the status of the project says:

[root@rjanik-openshift-merrn-1 ~]# oc get project
NAME                                DISPLAY NAME          STATUS
default                                                   Active
install-test                                              Terminating
kube-public                                               Active
...

Trying to clear the project by hand (not forced):
[root@rjanik-openshift-merrn-1 ~]# oc delete project install-test
Error from server (Conflict): Operation cannot be fulfilled on namespaces "install-test": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

If you need any info, let me know, but please keep in mind that I'm no OpenShift expert. If you can give me commands to run, that would be best.

OCP version:
[root@rjanik-openshift-merrn-1 ~]# oc version
oc v3.11.69
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://rjanik-openshift-merrn-1:8443
openshift v3.11.77
kubernetes v1.11.0+d4cacc0

Comment 31 daxuyin 2020-05-26 14:45:29 UTC
We're running OKD 3.11, it seems we have the same problem, after the command
$oc delete project <project name> --force --grace-period=0

we have error messages like these:
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "<project name>": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

There is no resource under this project:
$oc get all -n <project name>
No resource found

Does anyone know the resolution for this issue? Truly appreciate it.

Comment 32 Laercio 2020-06-04 21:40:13 UTC
I have the same problem. Here is 3.11.216 version.
I did a restart on the masters nodes and the problem continue.

[root@baham ~]# oc project testedetran
Now using project "testedetran" on server "https://oshift-interno.detran.pa.gov.br:443".
[root@baham ~]# oc get all
No resources found.
[root@baham ~]# oc delete project testedetran --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "testedetran": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
[root@baham ~]# oc describe project testedetran
Name:			testedetran
Created:		9 hours ago
Labels:			<none>
Annotations:		openshift.io/sa.scc.mcs=s0:c10,c5
			openshift.io/sa.scc.supplemental-groups=1000100000/10000
			openshift.io/sa.scc.uid-range=1000100000/10000
Display Name:		<none>
Description:		<none>
Status:			Terminating
Node Selector:		<none>
Quota:			<none>
Resource limits:	<none>

Does anyone know the a workaround?

Thank you for support

Comment 33 Roman Kravtsov 2020-06-05 09:57:53 UTC
Hello, Laercio.

The situation is similar to yours. Maybe it will help you. 


romank@sigmakrv okd]$ oc get serviceinstance
NAME                       CLASS                                    PLAN      STATUS                      AGE
jenkins-persistent-j82vm   ClusterServiceClass/jenkins-persistent   default   ErrorPollingLastOperation   6d

[romank@sigmakrv okd]$ oc edit serviceinstance jenkins-persistent-j82vm
serviceinstance.servicecatalog.k8s.io/jenkins-persistent-j82vm edited

# Your need to delete "finalizers", see below
#  finalizers:
#  - kubernetes-incubator/service-catalog

[romank@sigmakrv okd]$ oc delete serviceinstance jenkins-persistent-j82vm
Error from server (NotFound): serviceinstances.servicecatalog.k8s.io "jenkins-persistent-j82vm" not found

[romank@sigmakrv okd]$ oc get serviceinstance
No resources found.
[romank@sigmakrv okd]$


After that, the project will be deleted.