Bug 1782601
| Summary: | CRI-O reporting container OOMKilled despite clean exit and no memory pressure | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | W. Trevor King <wking> | ||||||
| Component: | Node | Assignee: | Peter Hunt <pehunt> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Sunil Choudhary <schoudha> | ||||||
| Severity: | low | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 4.3.0 | CC: | aos-bugs, jokerman, pehunt, rphillips | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 4.3.z | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-06-03 03:30:41 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
W. Trevor King
2019-12-11 22:33:20 UTC
Created attachment 1644235 [details]
pod:container_memory_usage_bytes:sum{pod="cloud-credential-operator-57f58db849-tl2mf"}
memory usage in the nominally-OOMKilled pod
opened a PR: https://github.com/containers/conmon/pull/98 Checking in on where this is in the 4.3 pipe, conmon#98 went out in 2.0.8 [1]. Latest 4.3 RHCOS is 43.81.201912171100.0 [2]. That RHCOS has conmon 2.0.7 [3]. No RHCOS with 2.0.8 means it doesn't matter which RHCOS image is currently living in machine-os-content and that we are still not ON_QE here. [1]: https://github.com/containers/conmon/commits/v2.0.8 [2]: curl -s https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/builds.json | jq -r '.builds[0].id' [3]: $ curl -s https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/43.81.201912171100.0/x86_64/commitmeta.json | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "conmon") | .[2]' we had to revert the change as it introduced missing real OOMs: https://github.com/containers/conmon/pull/104 Oh, I was getting this confused with 82ef9c7 (#97) which went out in 2.0.7 and was not reverted. But yeah, looks like #98 actually went out 2.0.7 and was reverted in 2.0.8. I actually think this has to do with the conmon monitor incorrectly reporting. As of now, conmonmon simply checks if the conmon pid is still running, then kills the container and spoofs an oom if it's not. There exists the chance that a race could happen with kubelet, systemd and conmonmon, where: kubelet attempts to kill a pod when tearing down the cgroup, systemd kills conmon first (arbitrarily, the tearing down of the cgroup is not ordered AFAIK) immediately after conmon is killed, the conmonmon loop runs, and sees that the pid is gone, kills the container and reports OOM systemd tries to kill container pid, sees its not there and moves on silently kubelet sees the cgroup is torn down, and also sees that an "oom" happened, and then incorrectly reports it I am working on a fix in conmonmon that involves actually checking if an oom happened. I am now seeing this issue predates conmonmon, and is not about my comment above. Sorry for the noise Also some spurious OOMKilled discussion in bug 1800609 and bug 1801568. this should be fixed as of https://github.com/containers/conmon/releases/tag/v2.0.12, which has been in 4.3.z for a bit Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2256 |