Bug 2020746 - Namespace stuck terminating: Failed to delete all resource types, 1 remaining: unexpected items still remain in namespace
Summary: Namespace stuck terminating: Failed to delete all resource types, 1 remaining...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Peter Hunt
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-05 18:25 UTC by Peter Hunt
Modified: 2022-04-01 18:28 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2003206
Environment:
Last Closed: 2022-04-01 18:28:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Comment 1 Peter Hunt 2021-11-05 18:28:02 UTC 
In investigating https://bugzilla.redhat.com/show_bug.cgi?id=2003206, it has been found that the kubelet has it's own variation of this bug. To test this, one needs the CRI-O fixes in https://bugzilla.redhat.com/show_bug.cgi?id=2003206. From the must-gather:
```
Nov 05 13:41:32.889891 jetlag-bm8 hyperkube[2272695]: I1105 13:41:32.889837 2272695 kubelet_pods.go:1979] "Failed to reduce cpu time for pod pending volume cleanup" podUID=035a7426-64d2-4047-b00a-bb175be433db err="open /sys/fs/cgroup/hugetlb/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod035a7426_64d2_4047_b00a_bb175be433db.slice/hugetlb.2MB.limit_in_bytes: no such file or directory"
```
The kubelet is failing to do volume cleanup, which is stalling pod teardown.

I have cloned the bug because we'll want to keep https://bugzilla.redhat.com/show_bug.cgi?id=2003206 to track the progress of the CRI-O fixes/backports. I will attach the kubelet log
Comment 2 Peter Hunt 2021-11-05 19:08:38 UTC 
https://drive.google.com/file/d/1W43FDbsR5ZS5gjOzMg2VfMByDUobX9-8/view?usp=sharing is the kubelet logs

also note, this kubelet came from a 4.10 nightly, though it's possible it also affects older versions
Comment 3 Tom Sweeney 2022-01-06 16:17:47 UTC 
Not completed this sprint.
Note You need to log in before you can comment on or make changes to this bug.