Bug 1931938 - RFE: chromium control groups organization for systemd-oomd
Summary: RFE: chromium control groups organization for systemd-oomd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: chromium
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Tom "spot" Callaway
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1913794
TreeView+ depends on / blocked
 
Reported: 2021-02-23 15:46 UTC by Chris Murphy
Modified: 2021-04-01 00:52 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-01 00:52:25 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Chris Murphy 2021-02-23 15:46:46 UTC
escription of problem:

Fedora 34 change to use systemd-oomd system wide. Earlyoom worked on a per process basis, so it tended to SIGTERM/SIGKILL on a per tab basis. But sd-oomd works at the cgroup level, and all Chromium processes are currently located in the same scope. This means Chromium as a whole is subject to being killed off when it exceeds resource control limits.

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd


Version-Release number of selected component (if applicable):
final release version of Chromium for Fedora 34


Additional info:


$ systemctl --user status app-gnome-chromium\\x2dbrowser-2976.scope 
● app-gnome-chromium\x2dbrowser-2976.scope - Application launched by gnome-shell
     Loaded: loaded (/run/user/1000/systemd/transient/app-gnome-chromium\x2dbrowser-2976.scope; transient)
  Transient: yes
     Active: active (running) since Tue 2021-02-23 08:44:03 MST; 1min 23s ago
      Tasks: 303 (limit: 14169)
     Memory: 786.8M
        CPU: 59.594s
     CGroup: /user.slice/user-1000.slice/user/app.slice/app-gnome-chromium\x2dbrowser-2976.scope
             ├─2976 /usr/lib64/chromium-browser/chromium-browser --enable-plugins --enable-extensions --enable-user-scripts --enable-printing --enable-gpu-ra>
             ├─2986 /usr/lib64/chromium-browser/chromium-browser --type=zygote --no-zygote-sandbox
             ├─2987 /usr/lib64/chromium-browser/chromium-browser --type=zygote
             ├─2989 /usr/lib64/chromium-browser/chromium-browser --type=zygote
             ├─3075 /usr/lib64/chromium-browser/chromium-browser --type=gpu-process --field-trial-handle=3970503390997581230,10988921493145911832,131072 --en>
             ├─3077 /usr/lib64/chromium-browser/chromium-browser --type=utility --utility-sub-type=network.mojom.NetworkService --field-trial-handle=39705033>
             ├─3085 /usr/lib64/chromium-browser/chromium-browser --type=utility --utility-sub-type=storage.mojom.StorageService --field-trial-handle=39705033>
             ├─3113 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3280 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3330 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3349 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3362 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3376 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3389 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3402 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3415 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3429 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3442 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3460 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3473 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3495 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3511 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3546 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3560 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             ├─3580 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>
             └─3597 /usr/lib64/chromium-browser/chromium-browser --type=renderer --field-trial-handle=3970503390997581230,10988921493145911832,131072 --lang=>

Comment 1 Tom "spot" Callaway 2021-02-23 15:59:46 UTC
Okay. It's not clear what I should do here in response. Chromium doesn't launch from a systemd service/unit/socket/whatever.

Is this something I should try to prevent somehow, or is it acceptable that if Chromium runs away and eats all the memory, it gets OOM killed?

Comment 2 Anita Zhang 2021-03-02 09:45:19 UTC
GNOME and friends will launch most (all?) processes these days in its own systemd unit. It would be nice to further split the chromium processes in this unit into their own cgroup for better resource management.

I left a comment on the Firefox RFE that might also be applicable here (https://bugzilla.redhat.com/show_bug.cgi?id=1931934#c2) if chromium is interested in splitting its processes into their own cgroups. Fedora is switching to systemd-oomd which operates at the cgroup level so it would be nice to not have to kill the whole chromium unit with all processes if it can be avoided with individual cgroups.

Comment 3 Tom "spot" Callaway 2021-03-03 16:51:15 UTC
Without rewriting how Chromium works, since chromium is already launched from a shell script wrapper (/usr/bin/chromium), is there something I could do in there to split the chromium process (and its many many children) into its own cgroup (or cgroups)? It is not clear from the Feature Request how to do this, and the manpage for "systemd-run" is equally confusing, as it appears to refer to services.

Comment 4 Michael Catanzaro 2021-03-18 21:17:37 UTC
No, GNOME will already run apps in a separate cgroup, so there's nothing needed in your shell script.

The goal is to run Chromium's render processes in separate cgroups. (If Chromium has other untrusted processes that are allowed to crash, they should probably use separate cgroups too. But trusted subprocesses that are not expected to crash and cannot be killed independently should not use their own cgroups.) Anyway, to do that you need to modify the Chromium process launching code. It can be done by wrapping each subprocess in a systemd-run invocation. For WebKit, I wound up using: 'systemd-run --scope --slice-inherit --quiet --user'. To verify that it works, I run systemd-cgls and inspect the result. I will upload a patch in https://bugs.webkit.org/show_bug.cgi?id=223479 momentarily that may be used as inspiration, though I haven't looked at Chromium's process launching code, and I rather doubt it will be as simple for Chromium as it was for WebKit....

Comment 5 Chris Murphy 2021-03-19 06:46:28 UTC
Another possibility that doesn't require modifying every package, is easy to A/B test, as well as undo.
https://gitlab.freedesktop.org/benzea/cgroupify

More info in the Firefox version of the same RFE
https://bugzilla.redhat.com/show_bug.cgi?id=1931934#c3

Comment 6 Fedora Update System 2021-03-30 20:23:07 UTC
FEDORA-2021-af75ff35e7 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-af75ff35e7

Comment 7 Fedora Update System 2021-03-31 01:20:30 UTC
FEDORA-2021-af75ff35e7 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-af75ff35e7`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-af75ff35e7

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2021-04-01 00:52:25 UTC
FEDORA-2021-af75ff35e7 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.