Bug 1829942
| Summary: | Multiple OOMKs with 5.6.7. Never OOMed on 5.x, x < 6. Regression? | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Arcadiy Ivanov <arcadiy> | ||||
| Component: | kernel | Assignee: | systemd-maint | ||||
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 31 | CC: | airlied, bskeggs, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, lnykryn, masami256, mchehab, mjg59, msekleta, ssahani, s, steved, systemd-maint, y9t7sypezp, zbyszek | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-11-24 18:47:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Thanks for your report. There is usually a list of kernel modules at the end of a Call Trace.
Specifically, why is the kernel reporting that it is tainted?
[235793.944852] CPU: 3 PID: 1 Comm: systemd Tainted: G U OE 5.6.7-200.fc31.x86_64 #1
^^^^^^^
Can you reproduce the problem without any non-Fedora kernel modules?
It looks like systemd-journald triggered the oom-killer, but with an oom_score_adj of -250, another process was selected for termination.
Do you have an ABRT problem report for it?
[235795.348134] systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Terminating.
[235805.287209] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
^^^^^^^^^^^
(In reply to Steve from comment #1) > Thanks for your report. There is usually a list of kernel modules at the end > of a Call Trace. > > Specifically, why is the kernel reporting that it is tainted? > > [235793.944852] CPU: 3 PID: 1 Comm: systemd Tainted: G U OE > 5.6.7-200.fc31.x86_64 #1 > ^^^^^^^ > > Can you reproduce the problem without any non-Fedora kernel modules? This is simple NVidia module and I'll see if I can. (In reply to Steve from comment #2) > It looks like systemd-journald triggered the oom-killer, but with an > oom_score_adj of -250, another process was selected for termination. > > Do you have an ABRT problem report for it? > > [235795.348134] systemd[1]: systemd-journald.service: State 'stop-watchdog' > timed out. Terminating. > [235805.287209] systemd[1]: systemd-journald.service: Main process exited, > code=dumped, status=6/ABRT > > ^^^^^^^^^^^ No I can't sorry, its not available anywhere, probably because abrt crashed as well. *** Bug 1831380 has been marked as a duplicate of this bug. *** Looking at the process list you are running a lot of big programs, but with 64GB of RAM that should not be a problem. Looking at the logs you provided, the clue seems to be the first line: [235793.944847] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 So it is not the kernel itself which is deciding it needs to run the oom-killer, it is systemd. I hope the systemd maintainers will have a better idea of when / why systemd is doing this, so lets move this over to systemd. (In reply to Hans de Goede from comment #6) > I hope the systemd maintainers will have a better idea of when / why systemd > is doing this, so lets move this over to systemd. I don't think it's systemd. I just was able to trigger it in the most bizarre way imaginable. Here is my swappiness: $ sysctl vm.swappiness vm.swappiness = 1 After I've encountered the above bug I've added 32GB of swap in addition to my 64GB of RAM. I was just minutes ago building customer's code and noticed in my KDE System Load Viewer swap bar sitting high. I scrolled over and 8GB of swap has been consumed building docker images with at the most 20GB of RAM being utilized. I go "wth?!" and `sudo swapoff -a` while the build is ongoing. Swapoff freezes. I'm looking at the graph of swap utilization and it's going down having swap `total` follow `used` as the `used` drops. But the `used` drops very slowly as if there is a huge pressure to keep things in swap. It takes tens of seconds for used swap to shed 2GB out of 8GB used. And then OOM killer kills the `swapoff` (!!!) and a whole bunch of browser tabs, and browser plugins and Slack... And the RAM utilization never exceeded 20GB, if that. I'm attaching the journal showing all the things killed and this time it wasn't systemd. Created attachment 1686110 [details]
Swapoff OOMKs half the processes with 40GB of RAM to spare
``` $ journalctl -xe | grep "invoked oom" May 07 05:24:05 ai-karellen-lap kernel: docker-squash invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 07 05:24:05 ai-karellen-lap kernel: Telegram invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 May 07 05:24:05 ai-karellen-lap kernel: slack invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 ``` Furthermore, I'm fairly certain that Telegram and Slack don't really "decide" to "invoke" OOMK. This message is a reminder that Fedora 31 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '31'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 31 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |
Machine has 64G of RAM and **never OOMed once** before for in a year with any kernel. Upgraded to 5.6.7, OOM fest with new OOM Killer. [235793.944847] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [235793.944852] CPU: 3 PID: 1 Comm: systemd Tainted: G U OE 5.6.7-200.fc31.x86_64 #1 [235793.944852] Hardware name: Dell Inc. Precision 5540/0V030K, BIOS 1.6.0 02/06/2020 [235793.944853] Call Trace: [235793.944860] dump_stack+0x66/0x90 [235793.944862] dump_header+0x4a/0x1e2 [235793.944864] oom_kill_process.cold+0xb/0x10 [235793.944865] out_of_memory+0x226/0x4c0 [235793.944867] __alloc_pages_slowpath+0xb32/0xdd0 [235793.944870] __alloc_pages_nodemask+0x2c9/0x310 [235793.944871] pagecache_get_page+0xbf/0x370 [235793.944873] filemap_fault+0x6cc/0xac0 [235793.944875] ? __mod_lruvec_state+0x3f/0xe0 [235793.944876] ? page_add_file_rmap+0x86/0x230 [235793.944878] ? alloc_set_pte+0x35d/0x650 [235793.944880] ? _cond_resched+0x15/0x30 [235793.944881] ? down_read+0xe/0x90 [235793.944912] __xfs_filemap_fault+0x5b/0x1e0 [xfs] [235793.944915] __do_fault+0x36/0x100 [235793.944917] __handle_mm_fault+0xfa5/0x15a0 [235793.944919] handle_mm_fault+0xc4/0x1f0 [235793.944922] do_user_addr_fault+0x1f9/0x450 [235793.944924] page_fault+0x3e/0x50 [235793.944926] RIP: 0033:0x7fd143034780 [235793.944930] Code: Bad RIP value. [235793.944930] RSP: 002b:00007ffcafe78538 EFLAGS: 00010246 [235793.944932] RAX: 00000000000002ec RBX: 0000000000000000 RCX: 0000559f09fef9e0 [235793.944933] RDX: 00000000000002ec RSI: 00007ffcafe78660 RDI: 0000000000000000 [235793.944933] RBP: 00007ffcafe78590 R08: 0000000000000000 R09: 000000000000000f [235793.944934] R10: 0000000000000000 R11: 0000000000000202 R12: 0000559f0a10cb30 [235793.944935] R13: 0000000000000000 R14: 00007ffcafe78660 R15: 0000559f09fef9e0 [235793.944937] Mem-Info: [235793.944941] active_anon:8303166 inactive_anon:1726355 isolated_anon:0 active_file:465 inactive_file:386 isolated_file:16 unevictable:5355321 dirty:0 writeback:0 unstable:0 slab_reclaimable:493067 slab_unreclaimable:249206 mapped:199039 shmem:10600504 pagetables:36912 bounce:0 free:155532 free_pcp:966 free_cma:0 [235793.944943] Node 0 active_anon:33212664kB inactive_anon:6905420kB active_file:1860kB inactive_file:1544kB unevictable:21421284kB isolated(anon):0kB isolated(file):64kB mapped:796156kB dirty:0kB writeback:0kB shmem:42402016kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [235793.944944] Node 0 DMA free:15904kB min:16kB low:28kB high:40kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [235793.944947] lowmem_reserve[]: 0 1590 63860 63860 63860 [235793.944948] Node 0 DMA32 free:250716kB min:1680kB low:3308kB high:4936kB reserved_highatomic:0KB active_anon:233320kB inactive_anon:174236kB active_file:12kB inactive_file:4kB unevictable:970060kB writepending:0kB present:1729404kB managed:1663668kB mlocked:0kB kernel_stack:20kB pagetables:352kB bounce:0kB free_pcp:260kB local_pcp:0kB free_cma:0kB [235793.944951] lowmem_reserve[]: 0 0 62269 62269 62269 [235793.944952] Node 0 Normal free:355508kB min:356000kB low:419764kB high:483528kB reserved_highatomic:2048KB active_anon:32979444kB inactive_anon:6731184kB active_file:1872kB inactive_file:1236kB unevictable:20451224kB writepending:0kB present:64954368kB managed:63772072kB mlocked:92kB kernel_stack:34108kB pagetables:147296kB bounce:0kB free_pcp:3604kB local_pcp:244kB free_cma:0kB [235793.944955] lowmem_reserve[]: 0 0 0 0 0 [235793.944956] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB [235793.944961] Node 0 DMA32: 1105*4kB (UME) 1191*8kB (UME) 1108*16kB (UME) 1001*32kB (UME) 628*64kB (UME) 419*128kB (UME) 214*256kB (UME) 73*512kB (UM) 1*1024kB (U) 0*2048kB 0*4096kB = 250716kB [235793.944965] Node 0 Normal: 9035*4kB (U) 12183*8kB (UEH) 7728*16kB (UE) 2979*32kB (UE) 0*64kB 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 354500kB [235793.944970] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [235793.944971] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [235793.944971] 10601730 total pagecache pages [235793.944973] 0 pages in swap cache [235793.944974] Swap cache stats: add 0, delete 0, find 0/0 [235793.944974] Free swap = 0kB [235793.944975] Total swap = 0kB [235793.944975] 16674941 pages RAM [235793.944976] 0 pages HighMem/MovableOnly [235793.944976] 312030 pages reserved [235793.944977] 0 pages cma reserved [235793.944977] 0 pages hwpoisoned [235793.944978] Tasks state (memory values in pages): [235793.944978] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [235793.944987] [ 748] 0 748 11968 1099 98304 0 -1000 systemd-udevd [235793.944991] [ 1177] 0 1177 7811 156 69632 0 -1000 auditd [235793.944993] [ 1198] 0 1198 78373 362 106496 0 0 ModemManager [235793.944994] [ 1199] 0 1199 594 18 40960 0 0 acpid [235793.944996] [ 1201] 0 1201 1261 83 53248 0 0 alsactl [235793.944998] [ 1204] 70 1204 7814 108 77824 0 0 avahi-daemon [235793.944999] [ 1207] 0 1207 2098 103 53248 0 0 bluetoothd [235793.945000] [ 1208] 0 1208 3002 57 57344 0 0 bumblebeed [235793.945001] [ 1212] 0 1212 83475 5356 167936 0 0 firewalld [235793.945003] [ 1231] 0 1231 2825 33 53248 0 0 mcelog [235793.945004] [ 1238] 0 1238 78010 226 106496 0 0 rngd [235793.945006] [ 1239] 172 1239 38244 74 61440 0 0 rtkit-daemon [235793.945008] [ 1240] 0 1240 2579 251 53248 0 0 smartd [235793.945009] [ 1241] 0 1241 65960 176 102400 0 0 gssproxy [235793.945010] [ 1242] 0 1242 63436 395 114688 0 0 sssd [235793.945011] [ 1243] 0 1243 75865 192 98304 0 0 thermald [235793.945013] [ 1244] 0 1244 100346 518 126976 0 0 udisksd [235793.945014] [ 1266] 81 1266 67222 224 57344 0 -900 dbus-broker-lau [235793.945016] [ 1269] 0 1269 120095 587 167936 0 0 abrtd [235793.945017] [ 1271] 994 1271 21762 109 61440 0 0 chronyd [235793.945018] [ 1283] 81 1283 2013 792 49152 0 -900 dbus-broker [235793.945020] [ 1304] 70 1304 7768 84 61440 0 0 avahi-daemon [235793.945021] [ 1310] 0 1310 63653 661 118784 0 0 sssd_be [235793.945022] [ 1323] 998 1323 488558 1815 241664 0 0 polkitd [235793.945024] [ 1423] 0 1423 115695 930 118784 0 0 accounts-daemon [235793.945025] [ 1625] 0 1625 206668 1137 192512 0 0 NetworkManager [235793.945026] [ 1683] 0 1683 63802 772 114688 0 0 cupsd [235793.945028] [ 1690] 0 1690 6516 299 77824 0 0 systemd-logind [235793.945029] [ 1694] 0 1694 4552 47 53248 0 0 atd [235793.945030] [ 1696] 0 1696 56434 214 77824 0 0 crond [235793.945031] [ 1700] 0 1700 1121 57 49152 0 0 kdm [235793.945033] [ 1702] 0 1702 538819 5120 425984 0 0 containerd [235793.945034] [ 1727] 0 1727 277736 83027 999424 0 0 Xorg [235793.945036] [ 1733] 982 1733 114715 522 114688 0 0 colord [235793.945038] [ 2031] 0 2031 3146 303 65536 0 0 wpa_supplicant [235793.945039] [ 2075] 0 2075 9572 275 77824 0 0 kdm [235793.945040] [ 3149] 0 3149 513463 9676 507904 0 -500 dockerd [235793.945042] [ 3219] 0 3219 3670538 459372 5218304 0 0 CrashPlanServic [235793.945044] [ 3683] 0 3683 237311 578 585728 0 0 abrt-dump-journ [235793.945045] [ 3684] 0 3684 238844 607 1191936 0 0 abrt-dump-journ [235793.945046] [ 3686] 0 3686 233232 578 860160 0 0 abrt-dump-journ [235793.945048] [ 3783] 1000 3783 4985 483 73728 0 0 systemd [235793.945049] [ 3788] 1000 3788 12291 876 86016 0 0 (sd-pam) [235793.945050] [ 3847] 1000 3847 635630 1723 196608 0 0 pulseaudio [235793.945052] [ 3849] 1000 3849 121724 1192 249856 0 0 kwalletd5 [235793.945053] [ 3850] 1000 3850 31043 206 106496 0 0 startkde [235793.945055] [ 3970] 1000 3970 69161 81 61440 0 0 dbus-broker-lau [235793.945056] [ 3971] 1000 3971 1865 634 53248 0 0 dbus-broker [235793.945057] [ 4007] 1000 4007 109612 110 77824 0 0 gpg-agent [235793.945059] [ 4151] 1000 4151 4277 193 73728 0 0 gconf-helper [235793.945060] [ 4152] 1000 4152 58210 139 86016 0 0 gconfd-2 [235793.945062] [ 4533] 1000 4533 1790 133 57344 0 0 ssh-agent [235793.945063] [ 4551] 1000 4551 559 25 40960 0 0 start_kdeinit [235793.945064] [ 4555] 1000 4555 70016 517 122880 0 0 kdeinit5 [235793.945065] [ 4561] 1000 4561 463815 5233 516096 0 0 kded5 [235793.945067] [ 4578] 1000 4578 120152 1205 253952 0 0 kaccess [235793.945068] [ 4580] 1000 4580 110974 693 192512 0 0 plasma_session [235793.945069] [ 4587] 1000 4587 124037 3415 270336 0 0 ksmserver [235793.945071] [ 4591] 1000 4591 108188 677 167936 0 0 kglobalaccel5 [235793.945072] [ 4601] 1000 4601 39045 172 69632 0 0 dconf-service [235793.945073] [ 4604] 1000 4604 106104 582 155648 0 0 kscreen_backend [235793.945075] [ 4612] 1000 4612 207754 8425 479232 0 0 kwin_x11 [235793.945076] [ 4614] 1000 4614 1806731 132453 2293760 0 0 plasmashell [235793.945077] [ 4616] 1000 4616 109833 623 172032 0 0 xembedsniproxy [235793.945078] [ 4620] 1000 4620 159340 1228 278528 0 0 polkit-kde-auth [235793.945080] [ 4629] 0 4629 113977 373 114688 0 0 upowerd [235793.945081] [ 4642] 1000 4642 110174 615 172032 0 0 gmenudbusmenupr [235793.945082] [ 4644] 1000 4644 153411 5175 364544 0 0 konsole [235793.945084] [ 4657] 1000 4657 186628 1393 245760 0 0 kactivitymanage [235793.945085] [ 4669] 1000 4669 76284 171 90112 0 0 at-spi-bus-laun [235793.945086] [ 4670] 1000 4670 153191 4338 356352 0 0 krfb [235793.945087] [ 4677] 1000 4677 67096 75 57344 0 0 dbus-broker-lau [235793.945089] [ 4678] 1000 4678 1302 80 45056 0 0 dbus-broker [235793.945090] [ 4683] 1000 4683 241646 9610 413696 0 0 konsole [235793.945091] [ 4688] 1000 4688 152894 4657 360448 0 0 konsole [235793.945092] [ 4690] 1000 4690 58293 2146 98304 0 0 bash [235793.945094] [ 4697] 1000 4697 58293 2147 102400 0 0 bash [235793.945095] [ 4710] 1000 4710 58293 2147 86016 0 0 bash [235793.945096] [ 4747] 1000 4747 153386 5151 368640 0 0 konsole [235793.945097] [ 4767] 1000 4767 58473 2326 98304 0 0 bash [235793.945099] [ 4777] 1000 4777 58293 2120 94208 0 0 bash [235793.945100] [ 4808] 1000 4808 154061 5843 360448 0 0 konsole [235793.945101] [ 4816] 1000 4816 58293 2121 98304 0 0 bash [235793.945103] [ 4819] 1000 4819 58293 2119 90112 0 0 bash [235793.945104] [ 4822] 1000 4822 58293 2121 98304 0 0 bash [235793.945105] [ 4835] 1000 4835 58359 2210 90112 0 0 bash [235793.945106] [ 4862] 1000 4862 58293 2150 86016 0 0 bash [235793.945108] [ 4890] 1000 4890 58293 2151 98304 0 0 bash [235793.945109] [ 4928] 1000 4928 175542 7231 409600 0 0 dolphin [235793.945110] [ 4929] 1000 4929 58293 2148 90112 0 0 bash [235793.945111] [ 4937] 1000 4937 58293 2147 90112 0 0 bash [235793.945113] [ 4955] 1000 4955 58293 2147 98304 0 0 bash [235793.945114] [ 5085] 1000 5085 174287 5961 389120 0 0 dolphin [235793.945116] [ 5106] 1000 5106 58293 2146 98304 0 0 bash [235793.945117] [ 5139] 1000 5139 58293 2146 94208 0 0 bash [235793.945118] [ 5186] 1000 5186 58293 2120 94208 0 0 bash [235793.945119] [ 5254] 1000 5254 58293 2121 98304 0 0 bash [235793.945120] [ 5293] 1000 5293 58293 2120 98304 0 0 bash [235793.945122] [ 5328] 1000 5328 58293 2121 81920 0 0 bash [235793.945123] [ 5466] 1000 5466 173295 6673 405504 0 0 dolphin [235793.945124] [ 5900] 1000 5900 174260 5790 397312 0 0 dolphin [235793.945125] [ 6067] 1000 6067 1054 111 49152 0 0 ksysguardd [235793.945126] [ 6870] 1000 6870 108444 690 180224 0 0 kiod5 [235793.945128] [ 6926] 1000 6926 76266 672 155648 0 0 file.so [235793.945129] [ 7073] 1000 7073 76266 694 155648 0 0 file.so [235793.945131] [ 7331] 1000 7331 1208794 107545 1576960 0 0 Telegram [235793.945132] [ 7830] 1000 7830 193259 906 167936 0 0 xdg-desktop-por [235793.945133] [ 7884] 1000 7884 166171 185 135168 0 0 xdg-document-po [235793.945135] [ 7914] 1000 7914 110714 109 98304 0 0 xdg-permission- [235793.945136] [ 7985] 1000 7985 153475 1283 172032 0 0 xdg-desktop-por [235793.945137] [ 8148] 1000 8148 40189 188 81920 0 0 at-spi2-registr [235793.945139] [ 8281] 1000 8281 114093 928 217088 0 0 xdg-desktop-por [235793.945141] [ 8497] 1000 8497 22123 149 61440 0 0 pipewire [235793.945142] [ 8829] 1000 8829 131868 216 118784 0 0 redshift [235793.945143] [ 8856] 993 8856 295682 47064 593920 0 0 geoclue [235793.945144] [ 9686] 1000 9686 124236 1306 270336 0 0 dolphin [235793.945146] [ 9689] 1000 9689 238679 1721 278528 0 0 org_kde_powerde [235793.945147] [ 9697] 1000 9697 111067 152 102400 0 0 agent [235793.945148] [ 9724] 1000 9724 357723 19180 2097152 0 0 slack [235793.945150] [ 9725] 1000 9725 694665 13277 1294336 0 0 skypeforlinux [235793.945151] [ 9929] 1000 9929 99495 1778 323584 0 0 slack [235793.945152] [ 9932] 1000 9932 99232 1678 303104 0 0 skypeforlinux [235793.945153] [ 9984] 1000 9984 99232 1690 176128 0 0 skypeforlinux [235793.945155] [ 9997] 1000 9997 99495 1784 212992 0 0 slack [235793.945156] [ 10109] 1000 10109 111719 237 106496 0 0 gvfsd [235793.945158] [ 10296] 1000 10296 129011 9255 520192 0 0 slack [235793.945159] [ 10326] 1000 10326 234521 3754 458752 0 0 skypeforlinux [235793.945161] [ 10329] 1000 10329 378428 9161 491520 0 0 skypeforlinux [235793.945162] [ 10350] 1000 10350 123463 7014 442368 0 0 slack [235793.945163] [ 10353] 1000 10353 1267254 42212 4395008 0 0 skypeforlinux [235793.945164] [ 11681] 1000 11681 148772 458 135168 0 0 gnome-keyring-d [235793.945166] [ 19534] 0 19534 61574 498 102400 0 0 veracrypt [235793.945167] [ 19536] 0 19536 131259 511 196608 0 0 veracrypt [235793.945173] [ 19582] 1000 19582 439786 116390 2592768 0 0 chromium-freewo [235793.945175] [ 19587] 1000 19587 53844 18 61440 0 0 cat [235793.945176] [ 19588] 1000 19588 53844 17 65536 0 0 cat [235793.945178] [ 19594] 1000 19594 139299 3372 524288 0 0 chromium-freewo [235793.945179] [ 19596] 1000 19596 139299 3376 299008 0 0 chromium-freewo [235793.945180] [ 19618] 1000 19618 189726 21002 880640 0 200 chromium-freewo [235793.945182] [ 19654] 1000 19654 3878628 19598 1191936 0 300 chromium-freewo [235793.945183] [ 19716] 1000 19716 1234109 10518 950272 0 300 chromium-freewo [235793.945184] [ 19786] 1000 19786 1230775 8620 888832 0 300 chromium-freewo [235793.945185] [ 19799] 1000 19799 3862956 16894 1081344 0 300 chromium-freewo [235793.945187] [ 19805] 1000 19805 1254265 17231 1216512 0 300 chromium-freewo [235793.945188] [ 19825] 1000 19825 1228727 6801 831488 0 300 chromium-freewo [235793.945189] [ 19838] 1000 19838 1221044 5245 778240 0 300 chromium-freewo [235793.945191] [ 19852] 1000 19852 1239098 12563 1015808 0 300 chromium-freewo [235793.945192] [ 19869] 1000 19869 1259564 15846 1232896 0 300 chromium-freewo [235793.945194] [ 19910] 1000 19910 3864681 17248 1060864 0 300 chromium-freewo [235793.945195] [ 19935] 1000 19935 1228468 6918 843776 0 300 chromium-freewo [235793.945196] [ 19967] 1000 19967 1229748 7854 876544 0 300 chromium-freewo [235793.945198] [ 19999] 1000 19999 3861865 15669 1056768 0 300 chromium-freewo [235793.945199] [ 20024] 1000 20024 1257094 16491 1200128 0 300 chromium-freewo [235793.945200] [ 20049] 1000 20049 1220529 5545 786432 0 300 chromium-freewo [235793.945202] [ 20063] 1000 20063 1220529 4944 770048 0 300 chromium-freewo [235793.945203] [ 20076] 1000 20076 1234103 9373 913408 0 300 chromium-freewo [235793.945204] [ 20091] 1000 20091 1244640 11449 1069056 0 300 chromium-freewo [235793.945205] [ 20148] 1000 20148 1226166 5370 811008 0 300 chromium-freewo [235793.945207] [ 20208] 1000 20208 1225395 5026 806912 0 300 chromium-freewo [235793.945208] [ 20314] 1000 20314 1237500 8068 970752 0 300 chromium-freewo [235793.945209] [ 20409] 1000 20409 1240240 9226 1028096 0 300 chromium-freewo [235793.945211] [ 20560] 1000 20560 1232315 11052 946176 0 300 chromium-freewo [235793.945212] [ 20980] 1000 20980 224273 4799 557056 0 200 chromium-freewo [235793.945213] [ 21143] 1000 21143 3440692 62175 3170304 0 300 chromium-freewo [235793.945215] [ 21197] 1000 21197 1267942 28359 1519616 0 300 chromium-freewo [235793.945216] [ 21236] 1000 21236 2320187 24698 1470464 0 300 chromium-freewo [235793.945218] [ 21270] 1000 21270 1273702 24082 1421312 0 300 chromium-freewo [235793.945219] [ 21285] 1000 21285 1274333 37479 1769472 0 300 chromium-freewo [235793.945220] [ 21305] 1000 21305 1254133 22142 1298432 0 300 chromium-freewo [235793.945221] [ 21322] 1000 21322 1265355 23025 1404928 0 300 chromium-freewo [235793.945223] [ 21343] 1000 21343 2317982 38109 1720320 0 300 chromium-freewo [235793.945224] [ 21372] 1000 21372 1249701 14014 1167360 0 300 chromium-freewo [235793.945226] [ 21396] 1000 21396 1275569 52587 1925120 0 300 chromium-freewo [235793.945227] [ 21467] 1000 21467 1220468 4911 753664 0 300 chromium-freewo [235793.945228] [ 21488] 1000 21488 1353747 27016 2260992 0 300 chromium-freewo [235793.945229] [ 21544] 1000 21544 1270479 24659 1671168 0 300 chromium-freewo [235793.945231] [ 21568] 1000 21568 1255298 16789 1486848 0 300 chromium-freewo [235793.945232] [ 21596] 1000 21596 1242273 13275 1118208 0 300 chromium-freewo [235793.945233] [ 21613] 1000 21613 1254302 20615 1429504 0 300 chromium-freewo [235793.945235] [ 21637] 1000 21637 1291261 33443 2457600 0 300 chromium-freewo [235793.945236] [ 21680] 1000 21680 2372360 48963 2617344 0 300 chromium-freewo [235793.945238] [ 21711] 1000 21711 1329744 54663 2740224 0 300 chromium-freewo [235793.945239] [ 21763] 1000 21763 1256455 20631 1290240 0 300 chromium-freewo [235793.945241] [ 24801] 1000 24801 1003896 48189 1155072 0 0 krunner [235793.945242] [ 28104] 1000 28104 54196 93 69632 0 0 pycharm.sh [235793.945243] [ 28146] 1000 28146 3464925 826494 8060928 0 0 java [235793.945244] [ 34369] 1000 34369 2354213 65007 2379776 0 300 chromium-freewo [235793.945246] [ 54546] 1000 54546 60986 933 98304 0 0 mc [235793.945247] [ 54548] 1000 54548 57699 1564 86016 0 0 bash [235793.945249] [ 82487] 1000 82487 1255149 26540 1318912 0 300 chromium-freewo [235793.945250] [ 105286] 1000 105286 1277815 39491 1687552 0 300 chromium-freewo [235793.945251] [ 105359] 1000 105359 1226933 7975 868352 0 300 chromium-freewo [235793.945253] [ 179963] 1000 179963 1261721 23969 1400832 0 300 chromium-freewo [235793.945254] [ 223297] 1000 223297 1229494 8567 856064 0 300 chromium-freewo [235793.945255] [ 233194] 1000 233194 1185818 368291 6438912 0 0 thunderbird [235793.945256] [ 234636] 1000 234636 6402862 377702 23171072 0 0 slack [235793.945258] [ 239305] 1000 239305 61123 1010 94208 0 0 mc [235793.945259] [ 239307] 1000 239307 57580 1437 90112 0 0 bash [235793.945261] [ 284092] 1000 284092 154007 6685 385024 0 0 kwrite [235793.945262] [ 285799] 1000 285799 337616 16153 569344 0 0 VirtualBox [235793.945263] [ 285818] 1000 285818 63232 603 114688 0 0 VBoxXPCOMIPCD [235793.945264] [ 285824] 1000 285824 316278 2548 241664 0 0 VBoxSVC [235793.945266] [ 292386] 1000 292386 114504 329 126976 0 0 gvfsd-http [235793.945267] [ 313493] 1000 313493 60764 696 98304 0 0 mc [235793.945269] [ 313498] 1000 313498 57579 1437 86016 0 0 bash [235793.945270] [ 317429] 1000 317429 72582 633 147456 0 0 file.so [235793.945271] [ 415640] 1000 415640 61039 926 102400 0 0 mc [235793.945273] [ 415642] 1000 415642 57698 1566 90112 0 0 bash [235793.945274] [ 456698] 1000 456698 2307840 29410 1515520 0 300 chromium-freewo [235793.945275] [ 458487] 1000 458487 1879 266 45056 0 0 fsnotifier64 [235793.945277] [ 463480] 1000 463480 1232942 10545 937984 0 300 chromium-freewo [235793.945278] [ 465064] 1000 465064 1246404 17333 1159168 0 300 chromium-freewo [235793.945279] [ 465139] 1000 465139 1284013 23565 1671168 0 300 chromium-freewo [235793.945281] [ 470242] 1000 470242 146428 7054 348160 0 0 spectacle [235793.945282] [ 477284] 1000 477284 1242239 13163 1150976 0 300 chromium-freewo [235793.945283] [ 485777] 1000 485777 1288152 53822 1961984 0 300 chromium-freewo [235793.945285] [ 488028] 1000 488028 1232825 8454 925696 0 300 chromium-freewo [235793.945286] [ 490123] 1000 490123 1242200 15484 1110016 0 300 chromium-freewo [235793.945287] [ 490181] 1000 490181 1242191 14956 1081344 0 300 chromium-freewo [235793.945288] [ 490219] 1000 490219 1235869 10961 970752 0 300 chromium-freewo [235793.945290] [ 490249] 1000 490249 2363183 64118 2379776 0 300 chromium-freewo [235793.945291] [ 504188] 1000 504188 1230517 7061 864256 0 300 chromium-freewo [235793.945292] [ 557816] 1000 557816 1227777 8008 856064 0 300 chromium-freewo [235793.945293] [ 558553] 1000 558553 57698 1566 69632 0 0 bash [235793.945295] [ 559944] 1000 559944 6687 374 94208 0 0 cmake [235793.945296] [ 559945] 1000 559945 14452 12943 155648 0 0 ninja-build [235793.945297] [ 567118] 1000 567118 243672 6862 454656 0 0 kscreenlocker_g [235793.945299] [ 572859] 1000 572859 33790 2001 278528 0 0 clang++ [235793.945300] [ 572860] 1000 572860 152058 118408 1257472 0 0 clang-9 [235793.945302] [ 573399] 1000 573399 1227958 6836 847872 0 300 chromium-freewo [235793.945303] [ 573431] 1000 573431 1215856 4301 679936 0 300 chromium-freewo [235793.945305] [ 573642] 1000 573642 33790 2000 278528 0 0 clang++ [235793.945306] [ 573643] 1000 573643 63818 30748 544768 0 0 clang-9 [235793.945307] [ 573656] 1000 573656 54470 357 69632 0 0 sh [235793.945308] [ 573660] 1000 573660 33970 2156 262144 0 0 clang++ [235793.945310] [ 573664] 1000 573664 463237 408476 3346432 0 0 ld [235793.945311] [ 573792] 1000 573792 33790 2003 274432 0 0 clang++ [235793.945312] [ 573795] 1000 573795 59116 25918 495616 0 0 clang-9 [235793.945314] [ 573812] 1000 573812 33790 2001 274432 0 0 clang++ [235793.945315] [ 573820] 1000 573820 47041 14264 397312 0 0 clang-9 [235793.945316] [ 573832] 1000 573832 33790 2001 282624 0 0 clang++ [235793.945317] [ 573833] 1000 573833 50391 17263 417792 0 0 clang-9 [235793.945319] [ 573846] 1000 573846 33790 2001 266240 0 0 clang++ [235793.945320] [ 573852] 1000 573852 49180 16178 417792 0 0 clang-9 [235793.945321] [ 573854] 1000 573854 33790 2001 266240 0 0 clang++ [235793.945322] [ 573857] 1000 573857 33790 2001 278528 0 0 clang++ [235793.945324] [ 573863] 1000 573863 54363 21308 454656 0 0 clang-9 [235793.945325] [ 573867] 1000 573867 43689 10937 389120 0 0 clang-9 [235793.945327] [ 573870] 1000 573870 54198 113 69632 0 0 sh [235793.945328] [ 573871] 1000 573871 33793 2013 270336 0 0 clang++ [235793.945329] [ 573872] 1000 573872 350606 296054 2437120 0 0 ld [235793.945331] [ 573952] 1000 573952 72547 626 143360 0 0 file.so [235793.945332] [ 573972] 1000 573972 69821 345 143360 0 0 kdeinit5 [235793.945334] [ 574088] 1000 574088 69850 354 110592 0 0 kdeinit5 [235793.945335] [ 574134] 1000 574134 20471 373 139264 0 0 klauncher [235793.945336] [ 574172] 0 574172 66760 289 155648 0 0 sssd_nss [235793.945338] [ 574193] 1000 574193 54197 85 69632 0 0 sh [235793.945339] [ 574235] 0 574235 36163 276 245760 0 -250 systemd-journal [235793.945342] [ 574369] 1000 574369 54218 190 69632 0 0 ar [235793.945344] [ 574410] 0 574410 3817 79 61440 0 0 systemd-coredum [235793.945345] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=chromium-freewo,pid=34369,uid=1000 [235793.945400] Out of memory: Killed process 34369 (chromium-freewo) total-vm:9416852kB, anon-rss:246904kB, file-rss:0kB, shmem-rss:13124kB, UID:1000 pgtables:2324kB oom_score_adj:300 [235794.039067] oom_reaper: reaped process 34369 (chromium-freewo), now anon-rss:0kB, file-rss:0kB, shmem-rss:12100kB [235795.348134] systemd[1]: systemd-journald.service: State 'stop-watchdog' timed out. Terminating. [235805.287209] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [235805.287298] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. [235808.085694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 5. [235808.086067] systemd[1]: Stopped Journal Service. [235810.158692] systemd[1]: Starting Journal Service...