Bug 2068699

Link to similar thread about i3/sway having problems, and where the change owners refuse to see this as an overall detrimental change: https://bugzilla.redhat.com/show_bug.cgi?id=1933494

Can you please file out a report at upstream in result cgroups support can be implemented?
https://github.com/mate-desktop/mate-session-manager/issues
Please give them the same detailed description like here.
In the meanwhile i will look how to disable systemd-oomd in mate desktop spin.
But sorry i am not sure if it is possible to do and how to do it.
Personal i nevser run in this issue with my 32GB RAM system.
Systemd-oomd i enabled and i never noticed X stops working. 

Is it possible to uninstall systemd-oomd and earlyoom needs to be installed, or how did you fix it in your installation?

It seems that preventing systemd-oomd-defaults from installation is the solution.
https://fedoraproject.org/wiki/Changes/EnableSystemdOomd#Should_spins_that_don.27t_put_processes_in_separate_cgroups_be_excluded_from_this_change.3F

```
Should spins that don't put processes in separate cgroups be excluded from this change?

That will be left up to the maintainers of those spins. Based on feedback, the current plan is to enable systemd-oomd with the specified configuration by default to minimize fragmentation on the Fedora install base (the Upgrade/Compatibility section as been updated to reflect this). A separate subpackage, "systemd-oomd-defaults", controls the policy for systemd-oomd and excluding it or removing it (and performing a systemctl daemon-reload) will prevent systemd-oomd from killing anything; without a policy systemd-oomd doesn't act. 
```

Can you please confirm that removing systemd-oomd-defaults fixes the problems for you.
Please enable systemd-oomd before you uninstall systemd-oomd.
I like to know if the TODO works with default unit state.

I can't test it for myself as i never ran in this issue.

And do you use earlyoom instead?

FEDORA-2022-b66b743517 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-b66b743517

FEDORA-2022-8d14f0b9ec has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8d14f0b9ec

FEDORA-2022-b66b743517 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-b66b743517`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-b66b743517

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-324adf3898 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-324adf3898`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-324adf3898

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Note: Update must be installed with "dnf update mate-desktop* --best --allowerasing" which will be remove systemd-oomd-defaults.

FEDORA-2022-8d14f0b9ec has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-8d14f0b9ec`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-8d14f0b9ec

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-324adf3898 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-324adf3898`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-324adf3898

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-b66b743517 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-b66b743517`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-b66b743517

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-324adf3898 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-b66b743517 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-8d14f0b9ec has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

The need to use "--best --allowerasing" means most users won't get it.
Wouldn't it be better to use an "obsoletes", which dnf enforces by default during updates?

Moreover, are we sure systemd-oomd will be domesticated by just removing systemd-oomd-defaults?
I'm not sure, because documentation says it would use it's compile-time defaults, as portrayed in /etc/systemd/oomd.conf.

And from Fedora-MATE_Compiz-Live-x86_64-36-20220506.n.0.iso :

[liveuser@localhost-live ~]$ cat /etc/systemd/oomd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the oomd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/oomd.conf' to display the full config.
#
# See oomd.conf(5) for details




Note: My Documentation Source is https://www.freedesktop.org/software/systemd/man/oomd.conf.html

========================================== Excerpt ========================================================
The default configuration is set during compilation, so configuration is only needed when it is necessary to deviate from those defaults. Initially, the main configuration file in /etc/systemd/ contains commented out entries showing the defaults as a guide to the administrator. Local overrides can be created by editing this file or by creating drop-ins, as described below. Using drop-ins for local configuration is recommended over modifications to the main configuration file.
==================================================================================================

Sorry. I mistakenly deleted the most important bit... This is the output, complete with the compile-time defauls:

[liveuser@localhost-live ~]$ cat /etc/systemd/oomd.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the oomd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/oomd.conf' to display the full config.
#
# See oomd.conf(5) for details

[OOM]
#SwapUsedLimit=90%
#DefaultMemoryPressureLimit=60%
#DefaultMemoryPressureDurationSec=30s

(In reply to Davide Repetto from comment #16)
> The need to use "--best --allowerasing" means most users won't get it.
> Wouldn't it be better to use an "obsoletes", which dnf enforces by default
> during updates?
> 
Using a conflict is the correct way in fedora to uninstall another package.
And with a new installation you won't get this update question.
So you will see it it only one time when updating an existing installation.

> Moreover, are we sure systemd-oomd will be domesticated by just removing
> systemd-oomd-defaults?
> I'm not sure, because documentation says it would use it's compile-time
> defaults, as portrayed in /etc/systemd/oomd.conf.

See https://fedoraproject.org/wiki/Changes/EnableSystemdOomd#Should_spins_that_don.27t_put_processes_in_separate_cgroups_be_excluded_from_this_change.3F
```
 A separate subpackage, "systemd-oomd-defaults", controls the policy for systemd-oomd and excluding it or removing it (and performing a systemctl daemon-reload) will prevent systemd-oomd from killing anything; without a policy systemd-oomd doesn't act.
```
With a previous update i set a present for earlyoom, means it is enable when installed. Tip from
https://bugzilla.redhat.com/show_bug.cgi?id=1933494#c23 

Well, if you have sufficient memory it is save to uninstall earlyoom, which is possible with last setup.
For my 32GB Ram box i don't use a out-of-memory killer.
```
Recommends:  earlyoom
Conflicts: systemd-oomd-defaults
```
At least the right fix for this problem should be adding cgroup support to mate-desktop, to avoid that systemd-oomd will kill the whole x-session.
There is a upstream report. https://github.com/mate-desktop/mate-desktop/issues/519

This message is a reminder that Fedora Linux 34 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '34'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 34 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

*** Bug 2094477 has been marked as a duplicate of this bug. ***

This bug appears to have been reported against 'rawhide' during the Fedora Linux 37 development cycle.
Changing version to 37.

*** Bug 2139656 has been marked as a duplicate of this bug. ***

*** Bug 2143151 has been marked as a duplicate of this bug. ***

(In reply to Wolfgang Ulbrich from comment #3)
...
> I like to know if the TODO works with default unit state.
> I can't test it for myself as i never ran in this issue.

A stress test can be used to induce an out-of-memory condition. The Fedora stress-ng command has numerous options for consuming memory.

For example, there are seven "--malloc..." options. A search of the stress-ng man page for "memory" will find many more options.

Caution: A stress test can make your system unusable or even cause it to crash, so it should only be run in a test environment.

(In reply to Davide Repetto from comment #16)
> The need to use "--best --allowerasing" means most users won't get it.

I am an experienced Linux user, yet I wasted a lot of time analyzing and reporting a duplicate of Bug 2143151.

So the current solution of using special dnf options is only a *workaround*. Specifically, having to use the "--skip-broken" option tells most people that there is something wrong. I suggest documenting this on the common issues page:

https://ask.fedoraproject.org/tags/c/common-issues/141/f37

Adding Kamil Páral to CC list.

Kamil: It seems like this should be added to the Common Issues page, because it can affect many MATE users, and the proposed workaround is not intuitive.

Sounds like a good candidate for Common Issues. Somebody please create the issue description in the Proposed Common Issues category, following the template (look at other Common Issues for inspiration how it should look like). Thanks!
https://ask.fedoraproject.org/c/common-issues/common-issues-proposed/142

(In reply to Steve from comment #24)
> A stress test can be used to induce an out-of-memory condition.

A virtual machine can also be used to test out-of-memory handling.

With the Fedora Virtual Machine Manager (virt-manager), you can easily configure how much memory is allocated for the virtual machine.

https://virt-manager.org/

(In reply to Davide Repetto from comment #16)
> Moreover, are we sure systemd-oomd will be domesticated by just removing systemd-oomd-defaults?

That's not necessary. Systemd has a convention that allows packages or users to *override* default configuration files by installing "drop-in" configuration files in certain well-defined directories.

Further, default configuration files can be disabled by creating a symlink with the same name as the default configuration file that points to /dev/null.

The oomd.conf man page explains those alternatives. This seems like the simplest approach:

"To disable a configuration file supplied by the vendor, the recommended way is to place a symlink to /dev/null in the configuration directory in /etc/, with the same filename as the vendor configuration file."

That says "/etc/", but that doesn't seem to be completely correct.

I believe the "drop-in" configuration files would go in /etc/systemd/*.conf.d/.

Thus, the MATE installer could create a directory named /etc/systemd/mate.conf.d/ and amend or disable configuration files from systemd-oomd-defaults as required.

That solution avoids removing the systemd-oomd-defaults package and is much more transparent.

(In reply to Steve from comment #28)
> Systemd has a convention that allows packages or users
> to *override* default configuration files by installing "drop-in"
> configuration files in certain well-defined directories.

The systemd-system.conf man page documents those directories in detail. And my previous recommendation appears to be inconsistent with the documentation:

"When packages need to customize the configuration, they can install drop-ins under /usr/. Files in /etc/ are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor packages."

Thus, the MATE systemd configuration files would go in /usr/lib/systemd/mate.conf.d/.

This works:

$ find /etc -type l -name \*oom\*.conf -exec 'ls' '-lF' '{}' '+' 2>/dev/null
lrwxrwxrwx. 1 root root 9 Nov 18 16:51 /etc/systemd/oomd.conf.d/10-oomd-defaults.conf -> /dev/null
lrwxrwxrwx. 1 root root 9 Nov 18 16:59 /etc/systemd/system/system.slice.d/10-oomd-per-slice-defaults.conf -> /dev/null
lrwxrwxrwx. 1 root root 9 Nov 18 16:59 /etc/systemd/system/user-.slice.d/10-oomd-per-slice-defaults.conf -> /dev/null
lrwxrwxrwx. 1 root root 9 Nov 18 17:00 /etc/systemd/user/slice.d/10-oomd-per-slice-defaults.conf -> /dev/null

$ oomctl
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 30s
System Context:
        Memory: Used: 1.1G Total: 3.8G
        Swap: Used: 0B Total: 3.8G
Swap Monitored CGroups:
Memory Pressure Monitored CGroups:

$ rpm -q systemd-oomd-defaults 
systemd-oomd-defaults-251.8-586.fc37.noarch

Tested with F37 Workstation in a VM.

Using stress-ng in a VM with 4G memory*, I cannot reproduce the alleged session-killing claimed in the original bug report.

After removing earlyoom, installing systemd-oomd-defaults**, and restarting***:

Start "top" in a terminal window and sort by memory usage with the "M" command.

In a second terminal window, run:

$ stress-ng --malloc 8 --malloc-bytes 1G
stress-ng: info:  [1966] defaulting to a 86400 second (1 day, 0.00 secs) run per stressor
stress-ng: info:  [1966] dispatching hogs: 8 malloc
^Cstress-ng: info:  [1966] successful run completed in 135.92s (2 mins, 15.92 secs)

The system journal shows that stress-ng processes are killed, but the whole MATE session is not:

$ journalctl --no-hostname -b -p err | fgrep -m 1 'Out of memory:'
Nov 22 11:29:06 kernel: Out of memory: Killed process 1981 (stress-ng) total-vm:1438262864kB, anon-rss:289616kB, file-rss:0kB, shmem-rss:12kB, UID:1000 pgtables:109536kB oom_score_adj:1000

Tested with:

$ rpm -qa mate-desktop systemd
mate-desktop-1.26.0-6.fc37.x86_64
systemd-251.8-586.fc37.x86_64

$ uname -r
6.0.9-300.fc37.x86_64

== Footnotes ==

* Verify with:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       569Mi       2.5Gi        19Mi       747Mi       3.0Gi
Swap:          3.8Gi          0B       3.8Gi

** This will bypass dependency checking:
# rpm -iv --nodeps systemd-oomd-defaults-251.8-586.fc37.noarch.rpm

*** Use "oomctl" to verify that the configuration from systemd-oomd-defaults has taken effect.

This command can be used to verify that the stress-ng command is running in the same cgroup as mate-session:

$ ps -e -o pid,comm,cgroup | egrep 'mate-session|stress-ng'
    980 mate-session    0::/user.slice/user-1000.slice/session-2.scope
   1968 stress-ng       0::/user.slice/user-1000.slice/session-2.scope
   1969 stress-ng       0::/user.slice/user-1000.slice/session-2.scope
   1970 stress-ng       0::/user.slice/user-1000.slice/session-2.scope
   1971 stress-ng       0::/user.slice/user-1000.slice/session-2.scope
   1972 stress-ng       0::/user.slice/user-1000.slice/session-2.scope

I just hit this issue on f36 (not sure why now and not earlier) while trying to do an update prior to system-upgrade to f37.

It took a bit of digging to find what was causing the conflict (intentional in mate-desktop). Not the best end user experience.

*** Bug 2168236 has been marked as a duplicate of this bug. ***

FEDORA-2023-2a193c0384 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a193c0384

FEDORA-2023-82251ee7c9 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-82251ee7c9

FEDORA-2023-2a193c0384 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2a193c0384

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-82251ee7c9 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-82251ee7c9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-82251ee7c9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-82251ee7c9 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2023-82251ee7c9

FEDORA-2023-82251ee7c9 has been pushed to the Fedora 37 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-82251ee7c9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-82251ee7c9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-82251ee7c9 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Looking at this for common issues, it seems like in current F36, F37 and F38 comps, systemd-oomd-defaults is not in any group that would be included in Xfce: it's only in workstation-product , cloud-server , gnome-desktop , kde-desktop and server-product . However, at the time F36 was released, it was in @core. It was also in @core for F34 and F35. So this issue will only affect folks who installed F34, F35 or F36 and upgraded, I think.

I think the conflicts that were added to mate-desktop-configs for some time should already alerted folks affected sufficiently, so I'm not sure there's really much value in documenting this at common issues now, so I'll remove the tag.

This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

I am thinking about to change `Recommends:  earlyoom` to `Recommends:  systemd-oomd-defaults` for f40, to get closer to common fedora.
In the end only one user was affected  by this issue and more users was want to use systemd-oomd-defaults.
And it is possible to uninstall systemd-oomd-defaults after it was removed from core group.