Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2020746

Summary:	Namespace stuck terminating: Failed to delete all resource types, 1 remaining: unexpected items still remain in namespace
Product:	OpenShift Container Platform	Reporter:	Peter Hunt <pehunt>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	Kubelet	QA Contact:	Sunil Choudhary <schoudha>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	medium
Priority:	medium	CC:	akrzos, aos-bugs, harpatil, minmli, mmethot, nagrawal, tsweeney
Version:	4.10	Keywords:	Reopened
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2003206	Environment:
Last Closed:	2022-04-01 18:28:39 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 1 Peter Hunt 2021-11-05 18:28:02 UTC

In investigating https://bugzilla.redhat.com/show_bug.cgi?id=2003206, it has been found that the kubelet has it's own variation of this bug. To test this, one needs the CRI-O fixes in https://bugzilla.redhat.com/show_bug.cgi?id=2003206. From the must-gather:
```
Nov 05 13:41:32.889891 jetlag-bm8 hyperkube[2272695]: I1105 13:41:32.889837 2272695 kubelet_pods.go:1979] "Failed to reduce cpu time for pod pending volume cleanup" podUID=035a7426-64d2-4047-b00a-bb175be433db err="open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod035a7426_64d2_4047_b00a_bb175be433db.slice/hugetlb.2MB.limit_in_bytes: no such file or directory"
```
The kubelet is failing to do volume cleanup, which is stalling pod teardown.

I have cloned the bug because we'll want to keep https://bugzilla.redhat.com/show_bug.cgi?id=2003206 to track the progress of the CRI-O fixes/backports. I will attach the kubelet log

Comment 2 Peter Hunt 2021-11-05 19:08:38 UTC

https://drive.google.com/file/d/1W43FDbsR5ZS5gjOzMg2VfMByDUobX9-8/view?usp=sharing is the kubelet logs

also note, this kubelet came from a 4.10 nightly, though it's possible it also affects older versions

Comment 3 Tom Sweeney 2022-01-06 16:17:47 UTC

Not completed this sprint.