Bug 2074820
| Summary: | Pods left on node even though they're gone from control plane | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Andrei Neagoe <andrei> |
| Component: | Node | Assignee: | Peter Hunt <pehunt> |
| Node sub component: | CRI-O | QA Contact: | Sunil Choudhary <schoudha> |
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | unspecified | CC: | aos-bugs |
| Version: | 4.9 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-04-22 16:49:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I've collected a must-gather, available here: https://drive.google.com/file/d/1ehLH0WPnsG47n0CgO6IeOaCBRCaRlhfV/view?usp=sharing I've also collected worker logs for relevant units, available under host_service_logs/workers/ I am having trouble finding it in the must-gather, do you know the version of cri-o? Sure thing, please see below: crio version 1.22.1 Version: 1.22.1 GitCommit: 63ca93845d5fe05cdca826367afcb601ece8d7ad GitTreeState: clean BuildDate: 2022-02-08T09:11:33Z GoVersion: go1.16.13 Compiler: gc Platform: linux/amd64 Linkmode: dynamic BuildTags: seccomp, selinux SeccompEnabled: true AppArmorEnabled: false there's a new 1.22 version that just made it to stable ( I think), 1.22.3. It should have a fix to your problem. Is there an upgrade available? Not for OKD I'm afraid, as per https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/. I have now a daemon set in place that's reaping orphaned pods, which seems to be working fine. So as long as there's some fix somewhere along the way, I'm not in a rush anymore. Can you point me to the upstream ticket/bug report? Just curious to understand the root-cause... it was still not clear to me whether this issue is coming from crio itself or from kubelet or OpenShiftSDN. https://github.com/cri-o/cri-o/pull/5336 was the code change, which actually came in 1.22.2. Basically, previously cri-o removed the namespaces for the pod when the pod is removed. Removing the network namespace also removes the veth entries. Kubelet only actually removes the pod when the node hits kubelet GC percentage. We moved the namespace removal to sandbox stop, which happens much earlier (whenever all the containers are terminated). which made the situation not look like it's leaking We can go ahead and close this if the problem was already addressed. I will be able to test this once I upgrade to 4.10 (waiting now for a more recent release) and if required will follow-up. |
Description of problem: We've ran into an issue where we noticed a large number of veth interfaces (1000+) present on a node. Upon closer inspection, it seems that some pods are not correctly terminated and removed. Even though running processes are gone (ie no compute/memory resource consumption), this issue leads to waste of interfaces and IP address space as they're still assigned. The worst case we've observed was a node with over 5000 pods and equivalent veth interfaces. These accumulate slowly and so far I've been unable to reproduce artificially on a separate, test node. Version-Release number of selected component (if applicable): 4.9.0-0.okd-2022-02-12-140851, running with OpenShift SDN in multitenant mode. How reproducible: Create and destroy large numbers of pods Additional info: On affected nodes, there are pods listed under crictl pods which are in both Ready and NotReady state, that are unknown to the apiserver (ie kubectl get pods -A doesn't match any of them). Most of them are in NotReady state, in some cases since weeks. Found by observing large numbers of veth interfaces that were not present in vswitch database: [root@worker-12 ~]# ovs-vsctl list-ports br0 | grep veth | wc -l 42 [root@worker-12 ~]# ip a | grep veth | wc -l 2807 Previous version, 4.8.0-0.okd-2021-11-14-052418, did not show any signs of having such issues.