Bug 1974763

Summary: systemd-omg kills processes when when there is plenty of RAM available
Product: [Fedora] Fedora Reporter: Alex G. <mr.nuke.me>
Component: systemdAssignee: Anita Zhang <the.anitazha>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: fedoraproject, filbranden, flepied, kasong, lnykryn, msekleta, ssahani, s, systemd-maint, yuwatana, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-249.4-1.fc35 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-13 15:51:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex G. 2021-06-22 13:47:23 UTC
Description of problem:

So, I came back to my workstation today to start my work day. I was surprised to find that all my konsole windows that I had left open the previous day had mysteriously vanished. After lots of intervestigation, I discovered that systemd-omg had snuck in after midnight and murdered them. This was the only evidence left behind:

Jun 22 01:44:09 nuclearis3.gtech systemd-oomd[2755]: Killed /user.slice/user-1000.slice/user/app.slice/app-org.kde.konsole-0aa05c341ada4325b46b1791f3441295.scope due to swap used (7731081216) / total (8589930496) being more than 90.00%

This is problematic, because available memory , as shown by free -h, was above 10 GiB throughout the entire ordeal.

Version-Release number of selected component (if applicable):

systemd-248-2.fc34.x86_64


How reproducible:

Intermittent. This is the second time I've noticed it in 30 days.


Steps to Reproduce:
1. Launch lots of Konsole windows
2. Launch a background task that malloc()s and free()s large amounts of memory, but well within the available system memory.
3. Return the next day

Actual results:

Oooh, noo, where's all my windows?


Expected results:

Everything is like I left it.


Additional info:

System has 32 GiB of RAM. Plenty good for the programs I go out with.

Comment 1 Anita Zhang 2021-06-23 09:00:15 UTC
I will work on a patch to check available that available memory is also below a threshold before doing a swap-based kill. I received a separate report about this as well and there are a few situations where things tend to stick around in swap instead of going back to memory.

Comment 2 Alex G. 2021-06-23 21:43:48 UTC
That sounds great. Thank you!

Comment 3 Zbigniew Jędrzejewski-Szmek 2021-10-13 15:51:04 UTC
https://github.com/systemd/systemd/commit/cb5ce676d9
This was backported in v248.4.