Bug 1540779 - Project namespace stuck in terminating status
Summary: Project namespace stuck in terminating status
Keywords:
Status: CLOSED EOL
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Stefan Schimanski
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-01 00:54 UTC by Kenjiro Nakayama
Modified: 2022-03-13 14:40 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-07 11:21:50 UTC
Target Upstream Version:
Embargoed:
haowang: needinfo-


Attachments (Terms of Use)

Description Kenjiro Nakayama 2018-02-01 00:54:19 UTC
Description of problem:

- namespace stuck in "Terminating" status
- We confirmed that when we try to delete same error is seen as https://github.com/openshift/origin/issues/18125

  # oc delete project test2 --force
  Error from server (Conflict): Operation cannot be fulfilled on namespaces "test2": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

  # oc delete project test2 --force --grace-period=0
  warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
  Error from server (Conflict): Operation cannot be fulfilled on namespaces "test2": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

Version-Release number of selected component (if applicable):
- OCP 3.7

How reproducible: not 100%

Steps to Reproduce:
1. Not sure what is the trigger.

Actual results:

  # oc get projects -o wide | grep -i terminating
  test2                  test2                            Terminating
  test3                  test3                            Terminating

Expected results:
- namespace stuck in "Terminating" status

Additional info:
- As mentioned above, similar issue with https://github.com/openshift/origin/issues/18125

Comment 1 Kenjiro Nakayama 2018-02-01 05:17:09 UTC
Hi, as per upstream issue discussion, we checked resources remained in the project, but there are no such resources.

  # oc get all -n test2
  No resources found.

  # oc get project test2
  NAME      DISPLAY NAME   STATUS
  test2     test2          Terminating

Is there any workaround for this issue? If no workaround, we would like you to prioritize this ticket.

[1] https://github.com/openshift/origin/issues/18125

Comment 4 Dan Mace 2018-02-06 18:21:17 UTC
Kenjiro,

The /cases/02022112 directory seems to have disappeared from collab-shell.usersys.redhat.com.

I think the controller logs are the best diagnostic tool right now; could you try once more to provide the relevant logs? Thanks!

Comment 22 Richard Janík 2019-02-06 08:49:19 UTC
Hi,

we're running a 3.11 cluster and it looks like we hit something similar. We have an ansible playbook doing changes to an OCP cluster installed by an external tool, so the exact process may be hard to trace, if that's necessary. At one point, ansible attempts to run: oc delete project install-test , and this command has been stuck for 15+ hours. Logging in on the machine and investigating the status of the project says:

[root@rjanik-openshift-merrn-1 ~]# oc get project
NAME                                DISPLAY NAME          STATUS
default                                                   Active
install-test                                              Terminating
kube-public                                               Active
...

Trying to clear the project by hand (not forced):
[root@rjanik-openshift-merrn-1 ~]# oc delete project install-test
Error from server (Conflict): Operation cannot be fulfilled on namespaces "install-test": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

If you need any info, let me know, but please keep in mind that I'm no OpenShift expert. If you can give me commands to run, that would be best.

OCP version:
[root@rjanik-openshift-merrn-1 ~]# oc version
oc v3.11.69
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://rjanik-openshift-merrn-1:8443
openshift v3.11.77
kubernetes v1.11.0+d4cacc0

Comment 31 daxuyin 2020-05-26 14:45:29 UTC
We're running OKD 3.11, it seems we have the same problem, after the command
$oc delete project <project name> --force --grace-period=0

we have error messages like these:
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "<project name>": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

There is no resource under this project:
$oc get all -n <project name>
No resource found

Does anyone know the resolution for this issue? Truly appreciate it.

Comment 32 Laercio 2020-06-04 21:40:13 UTC
I have the same problem. Here is 3.11.216 version.
I did a restart on the masters nodes and the problem continue.

[root@baham ~]# oc project testedetran
Now using project "testedetran" on server "https://oshift-interno.detran.pa.gov.br:443".
[root@baham ~]# oc get all
No resources found.
[root@baham ~]# oc delete project testedetran --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "testedetran": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.
[root@baham ~]# oc describe project testedetran
Name:			testedetran
Created:		9 hours ago
Labels:			<none>
Annotations:		openshift.io/sa.scc.mcs=s0:c10,c5
			openshift.io/sa.scc.supplemental-groups=1000100000/10000
			openshift.io/sa.scc.uid-range=1000100000/10000
Display Name:		<none>
Description:		<none>
Status:			Terminating
Node Selector:		<none>
Quota:			<none>
Resource limits:	<none>

Does anyone know the a workaround?

Thank you for support

Comment 33 Roman Kravtsov 2020-06-05 09:57:53 UTC
Hello, Laercio.

The situation is similar to yours. Maybe it will help you. 


romank@sigmakrv okd]$ oc get serviceinstance
NAME                       CLASS                                    PLAN      STATUS                      AGE
jenkins-persistent-j82vm   ClusterServiceClass/jenkins-persistent   default   ErrorPollingLastOperation   6d

[romank@sigmakrv okd]$ oc edit serviceinstance jenkins-persistent-j82vm
serviceinstance.servicecatalog.k8s.io/jenkins-persistent-j82vm edited

# Your need to delete "finalizers", see below
#  finalizers:
#  - kubernetes-incubator/service-catalog

[romank@sigmakrv okd]$ oc delete serviceinstance jenkins-persistent-j82vm
Error from server (NotFound): serviceinstances.servicecatalog.k8s.io "jenkins-persistent-j82vm" not found

[romank@sigmakrv okd]$ oc get serviceinstance
No resources found.
[romank@sigmakrv okd]$


After that, the project will be deleted.


Note You need to log in before you can comment on or make changes to this bug.