systemd-238-11.gita76ee90.fc28.x86_64 VIRT in htop is displayed with 24.5 GB RES 170 MB after just one day utpime with "RuntimeMaxUse=10M" frankly on VMs with 768 MB RAM that leads to OOM situations [root@arrakis:~]$ cat /etc/systemd/journald.conf [Journal] Storage=volatile Compress=yes RateLimitInterval=10s RateLimitBurst=5000 RuntimeMaxUse=10M ForwardToSyslog=no [root@localhost:~]$ systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Drop-In: /etc/systemd/system/systemd-journald.service.d └─hidepid.conf, security.conf Active: active (running) since Sun 2019-01-13 13:13:06 CET; 1 day 1h ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 1934297 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 512) Memory: 169.5M CGroup: /system.slice/systemd-journald.service └─1934297 /usr/lib/systemd/systemd-journald
see also https://bugzilla.redhat.com/show_bug.cgi?id=1665931 no upstream point releases and cherry picking simply don't work on a large scale - it's that easy
> see also https://bugzilla.redhat.com/show_bug.cgi?id=1665931 That's this bug. Please post the /proc/PID/maps files for journald. Without that, it's to say anything. Also 'ls -l /var/log/journal/*/'.
see also should have been https://github.com/systemd/systemd/issues/9141#issuecomment-454130943 rebootet the VM at night because of a kernel-update and after 14 hours uptime it's alread at 13.5 gB VIRT and 101 MB RES in "systemctl status systemd-journald" there is nothing below /var/log/journal/ because of "Storage=volatile" and "RuntimeMaxUse=10M" which for me pretty clear means after rotate and in general it should not consume above 100 MB RAM [root@localhost:~]$ cat /etc/systemd/journald.conf [Journal] Storage=volatile Compress=yes RateLimitInterval=10s RateLimitBurst=5000 RuntimeMaxUse=10M ForwardToSyslog=no ----------------------------------------- [root@localhost:~]$ systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Drop-In: /etc/systemd/system/systemd-journald.service.d └─hidepid.conf, security.conf Active: active (running) since Mon 2019-01-14 22:00:33 CET; 14h ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 396 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 512) Memory: 101.1M CGroup: /system.slice/systemd-journald.service └─396 /usr/lib/systemd/systemd-journald Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. [root@localhost:~]$ ls -l /proc/$(pidof systemd-journald)/fd/ insgesamt 0 lr-x------ 1 root root 64 2019-01-14 22:00 0 -> /dev/null l-wx------ 1 root root 64 2019-01-14 22:00 1 -> /dev/null lrwx------ 1 root root 64 2019-01-15 01:06 10 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 11 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 12 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 13 -> 'socket:[18692]' lrwx------ 1 root root 64 2019-01-15 01:06 14 -> 'anon_inode:[timerfd]' lrwx------ 1 root root 64 2019-01-15 01:06 16 -> 'socket:[15164]' lrwx------ 1 root root 64 2019-01-14 22:00 17 -> 'socket:[18712]' lrwx------ 1 root root 64 2019-01-15 01:06 18 -> 'socket:[18714]' lrwx------ 1 root root 64 2019-01-15 01:06 19 -> 'socket:[18748]' l-wx------ 1 root root 64 2019-01-14 22:00 2 -> /dev/null lrwx------ 1 root root 64 2019-01-15 01:06 20 -> 'socket:[15166]' lrwx------ 1 root root 64 2019-01-15 01:06 21 -> 'socket:[16199]' lrwx------ 1 root root 64 2019-01-14 22:00 22 -> 'socket:[15170]' lrwx------ 1 root root 64 2019-01-15 01:06 23 -> 'socket:[13995]' lrwx------ 1 root root 64 2019-01-14 22:00 24 -> 'socket:[15174]' lrwx------ 1 root root 64 2019-01-14 22:00 25 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system lrwx------ 1 root root 64 2019-01-15 01:06 26 -> 'socket:[15238]' lrwx------ 1 root root 64 2019-01-15 01:06 27 -> 'socket:[15242]' lrwx------ 1 root root 64 2019-01-15 01:06 28 -> 'socket:[19921]' lrwx------ 1 root root 64 2019-01-15 01:06 29 -> 'socket:[19923]' lrwx------ 1 root root 64 2019-01-14 22:00 3 -> 'socket:[19484]' lrwx------ 1 root root 64 2019-01-15 01:06 30 -> 'socket:[19925]' lrwx------ 1 root root 64 2019-01-15 01:06 31 -> 'socket:[24764]' lrwx------ 1 root root 64 2019-01-15 01:06 32 -> 'socket:[20736]' lrwx------ 1 root root 64 2019-01-15 01:06 33 -> 'socket:[20618]' lrwx------ 1 root root 64 2019-01-14 22:00 34 -> 'socket:[21706]' lrwx------ 1 root root 64 2019-01-15 01:06 35 -> 'socket:[16270]' lrwx------ 1 root root 64 2019-01-14 22:00 36 -> 'socket:[16272]' lrwx------ 1 root root 64 2019-01-14 22:00 37 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system.journal lrwx------ 1 root root 64 2019-01-15 01:06 4 -> 'socket:[19487]' lrwx------ 1 root root 64 2019-01-15 01:06 5 -> 'socket:[19474]' l-wx------ 1 root root 64 2019-01-15 01:06 6 -> /dev/kmsg lrwx------ 1 root root 64 2019-01-15 01:06 7 -> 'anon_inode:[eventpoll]' lrwx------ 1 root root 64 2019-01-15 01:06 8 -> /dev/kmsg lr-x------ 1 root root 64 2019-01-15 01:06 9 -> /proc/sys/kernel/hostname [root@localhost:~]$ pmap --extended $(pidof systemd-journald) 396: /usr/lib/systemd/systemd-journald Address Kbytes RSS Dirty Mode Mapping 0000556c15ccd000 132 132 0 r-x-- systemd-journald 0000556c15eed000 8 8 8 r---- systemd-journald 0000556c15eef000 4 4 4 rw--- systemd-journald 0000556c1783f000 14046012 60088 60088 rw--- [ anon ] 00007f6dd2c82000 4 0 0 ----- [ anon ] 00007f6dd2c83000 8192 8 8 rw--- [ anon ] 00007f6dd3483000 4 0 0 ----- [ anon ] 00007f6dd3484000 12296 16 16 rw--- [ anon ] 00007f6dd4689000 6156 12 12 rw--- [ anon ] 00007f6dd4e8d000 12312 24 24 rw--- [ anon ] 00007f6dd5b54000 1280 292 292 rw-s- system.journal 00007f6dd5c94000 8208 16 16 rw--- [ anon ] 00007f6dd6699000 2052 4 4 rw--- [ anon ] 00007f6dd689a000 1608 372 0 r-x-- libm-2.27.so 00007f6dd6a2c000 2048 0 0 ----- libm-2.27.so 00007f6dd6c2c000 4 4 4 r---- libm-2.27.so 00007f6dd6c2d000 4 4 4 rw--- libm-2.27.so 00007f6dd6c2e000 144 64 0 r-x-- libudev.so.1.6.10 00007f6dd6c52000 2048 0 0 ----- libudev.so.1.6.10 00007f6dd6e52000 4 4 4 r---- libudev.so.1.6.10 00007f6dd6e53000 4 4 4 rw--- libudev.so.1.6.10 00007f6dd6e54000 692 64 0 r-x-- libsepol.so.1 00007f6dd6f01000 2048 0 0 ----- libsepol.so.1 00007f6dd7101000 4 4 4 r---- libsepol.so.1 00007f6dd7102000 4 4 4 rw--- libsepol.so.1 00007f6dd7103000 8 0 0 rw--- [ anon ] 00007f6dd7105000 1524 104 0 r-x-- libunistring.so.2.1.0 00007f6dd7282000 2044 0 0 ----- libunistring.so.2.1.0 00007f6dd7481000 16 16 16 r---- libunistring.so.2.1.0 00007f6dd7485000 4 4 4 rw--- libunistring.so.2.1.0 00007f6dd7486000 268 64 0 r-x-- libpcap.so.1.9.0 00007f6dd74c9000 2048 0 0 ----- libpcap.so.1.9.0 00007f6dd76c9000 8 8 8 r---- libpcap.so.1.9.0 00007f6dd76cb000 4 4 4 rw--- libpcap.so.1.9.0 00007f6dd76cc000 56 56 0 r-x-- libjson-c.so.4.0.0 00007f6dd76da000 2044 0 0 ----- libjson-c.so.4.0.0 00007f6dd78d9000 4 4 4 r---- libjson-c.so.4.0.0 00007f6dd78da000 4 4 4 rw--- libjson-c.so.4.0.0 00007f6dd78db000 28 24 0 r-x-- libargon2.so.0 00007f6dd78e2000 2044 0 0 ----- libargon2.so.0 00007f6dd7ae1000 4 4 4 r---- libargon2.so.0 00007f6dd7ae2000 4 4 4 rw--- libargon2.so.0 00007f6dd7ae3000 128 116 0 r-x-- libgpg-error.so.0.25.0 00007f6dd7b03000 2044 0 0 ----- libgpg-error.so.0.25.0 00007f6dd7d02000 4 4 4 r---- libgpg-error.so.0.25.0 00007f6dd7d03000 4 4 4 rw--- libgpg-error.so.0.25.0 00007f6dd7d04000 332 108 0 r-x-- libdevmapper.so.1.02 00007f6dd7d57000 2044 0 0 ----- libdevmapper.so.1.02 00007f6dd7f56000 4 4 4 r---- libdevmapper.so.1.02 00007f6dd7f57000 12 12 8 rw--- libdevmapper.so.1.02 00007f6dd7f5a000 4 0 0 rw--- [ anon ] 00007f6dd7f5b000 24 20 0 r-x-- libuuid.so.1.3.0 00007f6dd7f61000 2044 0 0 ----- libuuid.so.1.3.0 00007f6dd8160000 4 4 4 r---- libuuid.so.1.3.0 00007f6dd8161000 4 0 0 rw--- [ anon ] 00007f6dd8162000 16 16 0 r-x-- libattr.so.1.1.2448 00007f6dd8166000 2048 0 0 ----- libattr.so.1.1.2448 00007f6dd8366000 4 4 4 r---- libattr.so.1.1.2448 00007f6dd8367000 4 0 0 rw--- [ anon ] 00007f6dd8368000 12 12 0 r-x-- libdl-2.27.so 00007f6dd836b000 2044 0 0 ----- libdl-2.27.so 00007f6dd856a000 4 4 4 r---- libdl-2.27.so 00007f6dd856b000 4 4 4 rw--- libdl-2.27.so 00007f6dd856c000 524 60 0 r-x-- libpcre2-8.so.0.7.1 00007f6dd85ef000 2044 0 0 ----- libpcre2-8.so.0.7.1 00007f6dd87ee000 4 4 4 r---- libpcre2-8.so.0.7.1 00007f6dd87ef000 4 4 4 rw--- libpcre2-8.so.0.7.1 00007f6dd87f0000 344 64 0 r-x-- libmount.so.1.1.0 00007f6dd8846000 2044 0 0 ----- libmount.so.1.1.0 00007f6dd8a45000 12 12 12 r---- libmount.so.1.1.0 00007f6dd8a48000 4 4 4 rw--- libmount.so.1.1.0 00007f6dd8a49000 4 0 0 rw--- [ anon ] 00007f6dd8a4a000 300 64 0 r-x-- libblkid.so.1.1.0 00007f6dd8a95000 2044 0 0 ----- libblkid.so.1.1.0 00007f6dd8c94000 20 20 20 r---- libblkid.so.1.1.0 00007f6dd8c99000 4 4 4 rw--- libblkid.so.1.1.0 00007f6dd8c9a000 4 0 0 rw--- [ anon ] 00007f6dd8c9b000 88 60 0 r-x-- liblz4.so.1.8.1 00007f6dd8cb1000 2044 0 0 ----- liblz4.so.1.8.1 00007f6dd8eb0000 4 4 4 r---- liblz4.so.1.8.1 00007f6dd8eb1000 4 0 0 rw--- [ anon ] 00007f6dd8eb2000 148 60 0 r-x-- liblzma.so.5.2.4 00007f6dd8ed7000 2048 0 0 ----- liblzma.so.5.2.4 00007f6dd90d7000 4 4 4 r---- liblzma.so.5.2.4 00007f6dd90d8000 4 0 0 rw--- [ anon ] 00007f6dd90d9000 112 44 0 r-x-- libidn2.so.0.3.4 00007f6dd90f5000 2044 0 0 ----- libidn2.so.0.3.4 00007f6dd92f4000 4 4 4 r---- libidn2.so.0.3.4 00007f6dd92f5000 4 0 0 rw--- [ anon ] 00007f6dd92f6000 176 168 0 r-x-- libseccomp.so.2.3.3 00007f6dd9322000 2048 0 0 ----- libseccomp.so.2.3.3 00007f6dd9522000 84 84 84 r---- libseccomp.so.2.3.3 00007f6dd9537000 4 4 4 rw--- libseccomp.so.2.3.3 00007f6dd9538000 24 24 0 r-x-- libip4tc.so.0.1.0 00007f6dd953e000 2048 0 0 ----- libip4tc.so.0.1.0 00007f6dd973e000 4 4 4 r---- libip4tc.so.0.1.0 00007f6dd973f000 4 4 4 rw--- libip4tc.so.0.1.0 00007f6dd9740000 1108 188 0 r-x-- libgcrypt.so.20.2.4 00007f6dd9855000 2044 0 0 ----- libgcrypt.so.20.2.4 00007f6dd9a54000 8 8 8 r---- libgcrypt.so.20.2.4 00007f6dd9a56000 20 20 20 rw--- libgcrypt.so.20.2.4 00007f6dd9a5b000 320 64 0 r-x-- libcryptsetup.so.12.3.0 00007f6dd9aab000 2048 0 0 ----- libcryptsetup.so.12.3.0 00007f6dd9cab000 4 4 4 r---- libcryptsetup.so.12.3.0 00007f6dd9cac000 8 8 8 rw--- libcryptsetup.so.12.3.0 00007f6dd9cae000 28 28 0 r-x-- libacl.so.1.1.2253 00007f6dd9cb5000 2048 0 0 ----- libacl.so.1.1.2253 00007f6dd9eb5000 4 4 4 r---- libacl.so.1.1.2253 00007f6dd9eb6000 4 4 4 rw--- libacl.so.1.1.2253 00007f6dd9eb7000 16 16 0 r-x-- libcap.so.2.25 00007f6dd9ebb000 2044 0 0 ----- libcap.so.2.25 00007f6dda0ba000 4 4 4 r---- libcap.so.2.25 00007f6dda0bb000 4 4 4 rw--- libcap.so.2.25 00007f6dda0bc000 28 28 0 r-x-- librt-2.27.so 00007f6dda0c3000 2044 0 0 ----- librt-2.27.so 00007f6dda2c2000 4 4 4 r---- librt-2.27.so 00007f6dda2c3000 4 4 4 rw--- librt-2.27.so 00007f6dda2c4000 1748 1296 0 r-x-- libc-2.27.so 00007f6dda479000 2048 0 0 ----- libc-2.27.so 00007f6dda679000 16 16 16 r---- libc-2.27.so 00007f6dda67d000 8 8 8 rw--- libc-2.27.so 00007f6dda67f000 16 12 12 rw--- [ anon ] 00007f6dda683000 100 100 0 r-x-- libpthread-2.27.so 00007f6dda69c000 2048 0 0 ----- libpthread-2.27.so 00007f6dda89c000 4 4 4 r---- libpthread-2.27.so 00007f6dda89d000 4 4 4 rw--- libpthread-2.27.so 00007f6dda89e000 16 4 4 rw--- [ anon ] 00007f6dda8a2000 92 88 0 r-x-- libgcc_s-8-20181215.so.1 00007f6dda8b9000 2044 0 0 ----- libgcc_s-8-20181215.so.1 00007f6ddaab8000 4 4 4 r---- libgcc_s-8-20181215.so.1 00007f6ddaab9000 4 4 4 rw--- libgcc_s-8-20181215.so.1 00007f6ddaaba000 152 128 0 r-x-- libselinux.so.1 00007f6ddaae0000 2044 0 0 ----- libselinux.so.1 00007f6ddacdf000 4 4 4 r---- libselinux.so.1 00007f6ddace0000 4 4 4 rw--- libselinux.so.1 00007f6ddace1000 8 4 4 rw--- [ anon ] 00007f6ddace3000 1904 1448 0 r-x-- libsystemd-shared-238.so 00007f6ddaebf000 2044 0 0 ----- libsystemd-shared-238.so 00007f6ddb0be000 564 64 64 r---- libsystemd-shared-238.so 00007f6ddb14b000 4 4 4 rw--- libsystemd-shared-238.so 00007f6ddb14c000 8 8 8 rw--- [ anon ] 00007f6ddb14e000 156 156 0 r-x-- ld-2.27.so 00007f6ddb21b000 1280 620 620 rw-s- system 00007f6ddb35b000 60 56 56 rw--- [ anon ] 00007f6ddb36e000 4 4 4 rw-s- system.journal 00007f6ddb370000 4 4 4 rw-s- system 00007f6ddb371000 4 4 4 rw-s- kernel-seqnum 00007f6ddb372000 8 8 8 rw--- [ anon ] 00007f6ddb374000 4 4 4 r---- ld-2.27.so 00007f6ddb375000 4 4 4 rw--- ld-2.27.so 00007f6ddb376000 4 4 4 rw--- [ anon ] 00007ffe18fb5000 132 28 28 rw--- [ stack ] 00007ffe18fef000 12 0 0 r---- [ anon ] 00007ffe18ff2000 8 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------- ------- ------- total kB 14172792 66924 61680 [root@localhost:~]$ journalctl --rotate [root@localhost:~]$ ls -l /proc/$(pidof systemd-journald)/fd/ insgesamt 0 lr-x------ 1 root root 64 2019-01-14 22:00 0 -> /dev/null l-wx------ 1 root root 64 2019-01-14 22:00 1 -> /dev/null lrwx------ 1 root root 64 2019-01-15 01:06 10 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 11 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 12 -> 'anon_inode:[signalfd]' lrwx------ 1 root root 64 2019-01-15 01:06 13 -> 'socket:[18692]' lrwx------ 1 root root 64 2019-01-15 01:06 14 -> 'anon_inode:[timerfd]' lrwx------ 1 root root 64 2019-01-14 22:00 15 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system.journal lrwx------ 1 root root 64 2019-01-15 01:06 16 -> 'socket:[15164]' lrwx------ 1 root root 64 2019-01-14 22:00 17 -> 'socket:[18712]' lrwx------ 1 root root 64 2019-01-15 01:06 18 -> 'socket:[18714]' lrwx------ 1 root root 64 2019-01-15 01:06 19 -> 'socket:[18748]' l-wx------ 1 root root 64 2019-01-14 22:00 2 -> /dev/null lrwx------ 1 root root 64 2019-01-15 01:06 20 -> 'socket:[15166]' lrwx------ 1 root root 64 2019-01-15 01:06 21 -> 'socket:[16199]' lrwx------ 1 root root 64 2019-01-14 22:00 22 -> 'socket:[15170]' lrwx------ 1 root root 64 2019-01-15 01:06 23 -> 'socket:[13995]' lrwx------ 1 root root 64 2019-01-14 22:00 24 -> 'socket:[15174]' lrwx------ 1 root root 64 2019-01-15 01:06 26 -> 'socket:[15238]' lrwx------ 1 root root 64 2019-01-15 01:06 27 -> 'socket:[15242]' lrwx------ 1 root root 64 2019-01-15 01:06 28 -> 'socket:[19921]' lrwx------ 1 root root 64 2019-01-15 01:06 29 -> 'socket:[19923]' lrwx------ 1 root root 64 2019-01-14 22:00 3 -> 'socket:[19484]' lrwx------ 1 root root 64 2019-01-15 01:06 30 -> 'socket:[19925]' lrwx------ 1 root root 64 2019-01-15 01:06 31 -> 'socket:[24764]' lrwx------ 1 root root 64 2019-01-15 01:06 32 -> 'socket:[20736]' lrwx------ 1 root root 64 2019-01-15 01:06 33 -> 'socket:[20618]' lrwx------ 1 root root 64 2019-01-14 22:00 34 -> 'socket:[21706]' lrwx------ 1 root root 64 2019-01-15 01:06 35 -> 'socket:[16270]' lrwx------ 1 root root 64 2019-01-14 22:00 36 -> 'socket:[16272]' lrwx------ 1 root root 64 2019-01-15 01:06 4 -> 'socket:[19487]' lrwx------ 1 root root 64 2019-01-15 01:06 5 -> 'socket:[19474]' l-wx------ 1 root root 64 2019-01-15 01:06 6 -> /dev/kmsg lrwx------ 1 root root 64 2019-01-15 01:06 7 -> 'anon_inode:[eventpoll]' lrwx------ 1 root root 64 2019-01-15 01:06 8 -> /dev/kmsg lr-x------ 1 root root 64 2019-01-15 01:06 9 -> /proc/sys/kernel/hostname [root@localhost:~]$ pmap --extended $(pidof systemd-journald) 396: /usr/lib/systemd/systemd-journald Address Kbytes RSS Dirty Mode Mapping 0000556c15ccd000 132 132 0 r-x-- systemd-journald 0000556c15eed000 8 8 8 r---- systemd-journald 0000556c15eef000 4 4 4 rw--- systemd-journald 0000556c1783f000 14050108 60108 60108 rw--- [ anon ] 00007f6dd2c82000 4 0 0 ----- [ anon ] 00007f6dd2c83000 8192 8 8 rw--- [ anon ] 00007f6dd3483000 4 0 0 ----- [ anon ] 00007f6dd3484000 12296 16 16 rw--- [ anon ] 00007f6dd4689000 6156 12 12 rw--- [ anon ] 00007f6dd4e8d000 12312 24 24 rw--- [ anon ] 00007f6dd5c94000 8208 16 16 rw--- [ anon ] 00007f6dd6559000 1280 56 56 rw-s- system.journal 00007f6dd6699000 2052 4 4 rw--- [ anon ] 00007f6dd689a000 1608 372 0 r-x-- libm-2.27.so 00007f6dd6a2c000 2048 0 0 ----- libm-2.27.so 00007f6dd6c2c000 4 4 4 r---- libm-2.27.so 00007f6dd6c2d000 4 4 4 rw--- libm-2.27.so 00007f6dd6c2e000 144 64 0 r-x-- libudev.so.1.6.10 00007f6dd6c52000 2048 0 0 ----- libudev.so.1.6.10 00007f6dd6e52000 4 4 4 r---- libudev.so.1.6.10 00007f6dd6e53000 4 4 4 rw--- libudev.so.1.6.10 00007f6dd6e54000 692 64 0 r-x-- libsepol.so.1 00007f6dd6f01000 2048 0 0 ----- libsepol.so.1 00007f6dd7101000 4 4 4 r---- libsepol.so.1 00007f6dd7102000 4 4 4 rw--- libsepol.so.1 00007f6dd7103000 8 0 0 rw--- [ anon ] 00007f6dd7105000 1524 104 0 r-x-- libunistring.so.2.1.0 00007f6dd7282000 2044 0 0 ----- libunistring.so.2.1.0 00007f6dd7481000 16 16 16 r---- libunistring.so.2.1.0 00007f6dd7485000 4 4 4 rw--- libunistring.so.2.1.0 00007f6dd7486000 268 64 0 r-x-- libpcap.so.1.9.0 00007f6dd74c9000 2048 0 0 ----- libpcap.so.1.9.0 00007f6dd76c9000 8 8 8 r---- libpcap.so.1.9.0 00007f6dd76cb000 4 4 4 rw--- libpcap.so.1.9.0 00007f6dd76cc000 56 56 0 r-x-- libjson-c.so.4.0.0 00007f6dd76da000 2044 0 0 ----- libjson-c.so.4.0.0 00007f6dd78d9000 4 4 4 r---- libjson-c.so.4.0.0 00007f6dd78da000 4 4 4 rw--- libjson-c.so.4.0.0 00007f6dd78db000 28 24 0 r-x-- libargon2.so.0 00007f6dd78e2000 2044 0 0 ----- libargon2.so.0 00007f6dd7ae1000 4 4 4 r---- libargon2.so.0 00007f6dd7ae2000 4 4 4 rw--- libargon2.so.0 00007f6dd7ae3000 128 116 0 r-x-- libgpg-error.so.0.25.0 00007f6dd7b03000 2044 0 0 ----- libgpg-error.so.0.25.0 00007f6dd7d02000 4 4 4 r---- libgpg-error.so.0.25.0 00007f6dd7d03000 4 4 4 rw--- libgpg-error.so.0.25.0 00007f6dd7d04000 332 108 0 r-x-- libdevmapper.so.1.02 00007f6dd7d57000 2044 0 0 ----- libdevmapper.so.1.02 00007f6dd7f56000 4 4 4 r---- libdevmapper.so.1.02 00007f6dd7f57000 12 12 8 rw--- libdevmapper.so.1.02 00007f6dd7f5a000 4 0 0 rw--- [ anon ] 00007f6dd7f5b000 24 20 0 r-x-- libuuid.so.1.3.0 00007f6dd7f61000 2044 0 0 ----- libuuid.so.1.3.0 00007f6dd8160000 4 4 4 r---- libuuid.so.1.3.0 00007f6dd8161000 4 0 0 rw--- [ anon ] 00007f6dd8162000 16 16 0 r-x-- libattr.so.1.1.2448 00007f6dd8166000 2048 0 0 ----- libattr.so.1.1.2448 00007f6dd8366000 4 4 4 r---- libattr.so.1.1.2448 00007f6dd8367000 4 0 0 rw--- [ anon ] 00007f6dd8368000 12 12 0 r-x-- libdl-2.27.so 00007f6dd836b000 2044 0 0 ----- libdl-2.27.so 00007f6dd856a000 4 4 4 r---- libdl-2.27.so 00007f6dd856b000 4 4 4 rw--- libdl-2.27.so 00007f6dd856c000 524 60 0 r-x-- libpcre2-8.so.0.7.1 00007f6dd85ef000 2044 0 0 ----- libpcre2-8.so.0.7.1 00007f6dd87ee000 4 4 4 r---- libpcre2-8.so.0.7.1 00007f6dd87ef000 4 4 4 rw--- libpcre2-8.so.0.7.1 00007f6dd87f0000 344 64 0 r-x-- libmount.so.1.1.0 00007f6dd8846000 2044 0 0 ----- libmount.so.1.1.0 00007f6dd8a45000 12 12 12 r---- libmount.so.1.1.0 00007f6dd8a48000 4 4 4 rw--- libmount.so.1.1.0 00007f6dd8a49000 4 0 0 rw--- [ anon ] 00007f6dd8a4a000 300 64 0 r-x-- libblkid.so.1.1.0 00007f6dd8a95000 2044 0 0 ----- libblkid.so.1.1.0 00007f6dd8c94000 20 20 20 r---- libblkid.so.1.1.0 00007f6dd8c99000 4 4 4 rw--- libblkid.so.1.1.0 00007f6dd8c9a000 4 0 0 rw--- [ anon ] 00007f6dd8c9b000 88 60 0 r-x-- liblz4.so.1.8.1 00007f6dd8cb1000 2044 0 0 ----- liblz4.so.1.8.1 00007f6dd8eb0000 4 4 4 r---- liblz4.so.1.8.1 00007f6dd8eb1000 4 0 0 rw--- [ anon ] 00007f6dd8eb2000 148 60 0 r-x-- liblzma.so.5.2.4 00007f6dd8ed7000 2048 0 0 ----- liblzma.so.5.2.4 00007f6dd90d7000 4 4 4 r---- liblzma.so.5.2.4 00007f6dd90d8000 4 0 0 rw--- [ anon ] 00007f6dd90d9000 112 44 0 r-x-- libidn2.so.0.3.4 00007f6dd90f5000 2044 0 0 ----- libidn2.so.0.3.4 00007f6dd92f4000 4 4 4 r---- libidn2.so.0.3.4 00007f6dd92f5000 4 0 0 rw--- [ anon ] 00007f6dd92f6000 176 168 0 r-x-- libseccomp.so.2.3.3 00007f6dd9322000 2048 0 0 ----- libseccomp.so.2.3.3 00007f6dd9522000 84 84 84 r---- libseccomp.so.2.3.3 00007f6dd9537000 4 4 4 rw--- libseccomp.so.2.3.3 00007f6dd9538000 24 24 0 r-x-- libip4tc.so.0.1.0 00007f6dd953e000 2048 0 0 ----- libip4tc.so.0.1.0 00007f6dd973e000 4 4 4 r---- libip4tc.so.0.1.0 00007f6dd973f000 4 4 4 rw--- libip4tc.so.0.1.0 00007f6dd9740000 1108 188 0 r-x-- libgcrypt.so.20.2.4 00007f6dd9855000 2044 0 0 ----- libgcrypt.so.20.2.4 00007f6dd9a54000 8 8 8 r---- libgcrypt.so.20.2.4 00007f6dd9a56000 20 20 20 rw--- libgcrypt.so.20.2.4 00007f6dd9a5b000 320 64 0 r-x-- libcryptsetup.so.12.3.0 00007f6dd9aab000 2048 0 0 ----- libcryptsetup.so.12.3.0 00007f6dd9cab000 4 4 4 r---- libcryptsetup.so.12.3.0 00007f6dd9cac000 8 8 8 rw--- libcryptsetup.so.12.3.0 00007f6dd9cae000 28 28 0 r-x-- libacl.so.1.1.2253 00007f6dd9cb5000 2048 0 0 ----- libacl.so.1.1.2253 00007f6dd9eb5000 4 4 4 r---- libacl.so.1.1.2253 00007f6dd9eb6000 4 4 4 rw--- libacl.so.1.1.2253 00007f6dd9eb7000 16 16 0 r-x-- libcap.so.2.25 00007f6dd9ebb000 2044 0 0 ----- libcap.so.2.25 00007f6dda0ba000 4 4 4 r---- libcap.so.2.25 00007f6dda0bb000 4 4 4 rw--- libcap.so.2.25 00007f6dda0bc000 28 28 0 r-x-- librt-2.27.so 00007f6dda0c3000 2044 0 0 ----- librt-2.27.so 00007f6dda2c2000 4 4 4 r---- librt-2.27.so 00007f6dda2c3000 4 4 4 rw--- librt-2.27.so 00007f6dda2c4000 1748 1296 0 r-x-- libc-2.27.so 00007f6dda479000 2048 0 0 ----- libc-2.27.so 00007f6dda679000 16 16 16 r---- libc-2.27.so 00007f6dda67d000 8 8 8 rw--- libc-2.27.so 00007f6dda67f000 16 12 12 rw--- [ anon ] 00007f6dda683000 100 100 0 r-x-- libpthread-2.27.so 00007f6dda69c000 2048 0 0 ----- libpthread-2.27.so 00007f6dda89c000 4 4 4 r---- libpthread-2.27.so 00007f6dda89d000 4 4 4 rw--- libpthread-2.27.so 00007f6dda89e000 16 4 4 rw--- [ anon ] 00007f6dda8a2000 92 88 0 r-x-- libgcc_s-8-20181215.so.1 00007f6dda8b9000 2044 0 0 ----- libgcc_s-8-20181215.so.1 00007f6ddaab8000 4 4 4 r---- libgcc_s-8-20181215.so.1 00007f6ddaab9000 4 4 4 rw--- libgcc_s-8-20181215.so.1 00007f6ddaaba000 152 128 0 r-x-- libselinux.so.1 00007f6ddaae0000 2044 0 0 ----- libselinux.so.1 00007f6ddacdf000 4 4 4 r---- libselinux.so.1 00007f6ddace0000 4 4 4 rw--- libselinux.so.1 00007f6ddace1000 8 4 4 rw--- [ anon ] 00007f6ddace3000 1904 1448 0 r-x-- libsystemd-shared-238.so 00007f6ddaebf000 2044 0 0 ----- libsystemd-shared-238.so 00007f6ddb0be000 564 64 64 r---- libsystemd-shared-238.so 00007f6ddb14b000 4 4 4 rw--- libsystemd-shared-238.so 00007f6ddb14c000 8 8 8 rw--- [ anon ] 00007f6ddb14e000 156 156 0 r-x-- ld-2.27.so 00007f6ddb35b000 60 56 56 rw--- [ anon ] 00007f6ddb36f000 4 4 4 rw-s- system.journal 00007f6ddb371000 4 4 4 rw-s- kernel-seqnum 00007f6ddb372000 8 8 8 rw--- [ anon ] 00007f6ddb374000 4 4 4 r---- ld-2.27.so 00007f6ddb375000 4 4 4 rw--- ld-2.27.so 00007f6ddb376000 4 4 4 rw--- [ anon ] 00007ffe18fb5000 132 28 28 rw--- [ stack ] 00007ffe18fef000 12 0 0 r---- [ anon ] 00007ffe18ff2000 8 4 0 r-x-- [ anon ] ffffffffff600000 4 0 0 r-x-- [ anon ] ---------------- ------- ------- ------- total kB 14175604 66084 60840 [root@localhost:~]$ systemctl status systemd-journald ● systemd-journald.service - Journal Service Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled) Drop-In: /etc/systemd/system/systemd-journald.service.d └─hidepid.conf, security.conf Active: active (running) since Mon 2019-01-14 22:00:33 CET; 14h ago Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 396 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 512) Memory: 101.2M CGroup: /system.slice/systemd-journald.service └─396 /usr/lib/systemd/systemd-journald
Created attachment 1520721 [details] dead symlinks in ls -l /proc/$(pidof systemd-journald)/fd/ ls -l /proc/$(pidof systemd-journald)/fd/ shows all the symlinks to "socket" and "anon" are broken
------------------------------------------------------------------------- Debian Security Advisory DSA-4367-2 security https://www.debian.org/security/ Salvatore Bonaccorso January 15, 2019 https://www.debian.org/security/faq ------------------------------------------------------------------------- Package : systemd The Qualys Research Labs reported that the backported security fixes shipped in DSA 4367-1 contained a memory leak in systemd-journald. This and an unrelated bug in systemd-coredump are corrected in this update. Note that as the systemd-journald service is not restarted automatically a restart of the service or more safely a reboot is advised. For the stable distribution (stretch), these problems have been fixed in version 232-25+deb9u8.
> ls -l /proc/$(pidof systemd-journald)/fd/ shows all the symlinks to "socket" and "anon" are broken That's expected, those are not real files. Those symlinks are always "broken". > The Qualys Research Labs reported that the backported security fixes > shipped in DSA 4367-1 contained a memory leak in systemd-journald. This > and an unrelated bug in systemd-coredump are corrected in this update. I haven't had any reports that there was a bug in the upstream patch. This text implies that the text was in the backport to systemd-232. Do you have the link to the change? Is this problem a regression you're observing with systemd-238-11.gita76ee90.fc28? I looked over the commits that were added again in -11, and I don't see any leak.
at least i can't remember memory growing that fast while it was too fat anyways, frankly when wahtever software which don't have to write anything to disk and is limited to 10 MB memory by it#s configuration a) grows to VIRTL in the meantime way over 30 GB and c) has a RES of far above 100 MB with gowing numbers even after a rotate (whatever is there to rotate in that setup) somehting is broken the majority of small VMs hat journald currently on peak of memory usage and there is no valid reason for more that lets say 30 MB in that config which is to be honest also too much, a proper basic OS (sshd, rsyslog, nothing else) needs 70 MB after boot BTW: what is the point of "always broken symlinks" - don't look like a proper design
(In reply to Zbigniew Jędrzejewski-Szmek from comment #6) > I haven't had any reports that there was a bug in the upstream patch. This > text implies that the text was in the backport to systemd-232. Do you have > the link to the change? Here is the systemd stretch git: https://salsa.debian.org/systemd-team/systemd/commits/stretch I assume it's d9c31850 from Jan 14, 2019. Hope that helps.
Created attachment 1521105 [details] tiny server after 1 day uptime you can argue what you want, that is a joke on a tiny server running nothing else then asterisk and named while named is auth-only and so no caching and don't run syslog at all the whole machine could easily live with the amount of memory journald eats
Thanks. There's also #1666690. Both bugs are the same, but they are not present in the systemd-238 in F28. I tried the reproducer provided, and it doesn't reproduce. So while the memory usage does look suspiciously high, it doesn't seem to be related to recent changes.
please take also a look at that graph https://github.com/systemd/systemd/issues/9141#issuecomment-455802336 amd maybe it's suspiciously high longer, i am not always looking at journald memorry usage because i expected "RuntimeMaxUse=10M" just to work anyways, on a VOIP machine with 768 MB RAM and 524 MB used having journald 110 MB RES and 25.4GB VIRT is simply not acceptable and writing cronjobs restart journald every few hours feels so 1990
This is considered fixed https://github.com/systemd/systemd/issues/11502 with commit https://github.com/msekletar/systemd/commit/8bb299ce155abc7cf87718d071b2a1fd9e64760d Harald, can you possibly test that commit based on F28 source?
only if somebody provides at least a wokring scratch build while VIRT is terrible high and needs to be fixed too i care more about 70 MB to 200 MB after a view days uptime which leads to oom-killer on machines just running an authoritative nameserver which is pretty fine with a 768 MB RAM virtual machine and other servcies like dedicated sftp-server only running sshd at large dnf-transactions without forcibly restart journald before
systemd-239-10.git4dc7dce.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fb1547321
systemd-239-11.git4dc7dce.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fb1547321
systemd-239-11.git4dc7dce.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
This apparently was only fixed on F29, not F28. Can this be put through before support closes?
CLOSED ERRATA: it's amazing that nobody cares about the reported fedora release all the time fact is that journalctl --rotate also don't work at all on F28 it just doubles the stuff below /run/log without cleanup anything