Bug 1665931 - systemd-journald.service: crazy memory usage (24.5 GB VIRT, 170 MB RES) RuntimeMaxUse=10M
Summary: systemd-journald.service: crazy memory usage (24.5 GB VIRT, 170 MB RES) Runti...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-01-14 13:45 UTC by Harald Reindl
Modified: 2019-05-08 13:01 UTC (History)
9 users (show)

Fixed In Version: systemd-239-11.git4dc7dce.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-11 01:57:25 UTC


Attachments (Terms of Use)
dead symlinks in ls -l /proc/$(pidof systemd-journald)/fd/ (214.80 KB, image/png)
2019-01-15 11:15 UTC, Harald Reindl
no flags Details
tiny server after 1 day uptime (7.67 KB, image/png)
2019-01-16 19:02 UTC, Harald Reindl
no flags Details

Description Harald Reindl 2019-01-14 13:45:27 UTC
systemd-238-11.gita76ee90.fc28.x86_64

VIRT in htop is displayed with 24.5 GB
RES 170 MB

after just one day utpime with "RuntimeMaxUse=10M"
frankly on VMs with 768 MB RAM that leads to OOM situations

[root@arrakis:~]$ cat /etc/systemd/journald.conf
[Journal]
Storage=volatile
Compress=yes
RateLimitInterval=10s
RateLimitBurst=5000
RuntimeMaxUse=10M
ForwardToSyslog=no


[root@localhost:~]$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
  Drop-In: /etc/systemd/system/systemd-journald.service.d
           └─hidepid.conf, security.conf
   Active: active (running) since Sun 2019-01-13 13:13:06 CET; 1 day 1h ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 1934297 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 512)
   Memory: 169.5M
   CGroup: /system.slice/systemd-journald.service
           └─1934297 /usr/lib/systemd/systemd-journald

Comment 1 Harald Reindl 2019-01-14 19:49:09 UTC
see also https://bugzilla.redhat.com/show_bug.cgi?id=1665931

no upstream point releases and cherry picking simply don't work on a large scale - it's that easy

Comment 2 Zbigniew Jędrzejewski-Szmek 2019-01-14 21:49:30 UTC
> see also https://bugzilla.redhat.com/show_bug.cgi?id=1665931
That's this bug.

Please post the /proc/PID/maps files for journald. Without that, it's to say anything.
Also 'ls -l /var/log/journal/*/'.

Comment 3 Harald Reindl 2019-01-15 11:13:08 UTC
see also should have been https://github.com/systemd/systemd/issues/9141#issuecomment-454130943

rebootet the VM at night because of a kernel-update and after 14 hours uptime it's alread at 13.5 gB VIRT and 101 MB RES in "systemctl status systemd-journald"

there is nothing below /var/log/journal/ because of "Storage=volatile" and "RuntimeMaxUse=10M" which for me pretty clear means after rotate and in general it should not consume above 100 MB RAM

[root@localhost:~]$ cat /etc/systemd/journald.conf
[Journal]
Storage=volatile
Compress=yes
RateLimitInterval=10s
RateLimitBurst=5000
RuntimeMaxUse=10M
ForwardToSyslog=no

-----------------------------------------

[root@localhost:~]$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
  Drop-In: /etc/systemd/system/systemd-journald.service.d
           └─hidepid.conf, security.conf
   Active: active (running) since Mon 2019-01-14 22:00:33 CET; 14h ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 396 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 512)
   Memory: 101.1M
   CGroup: /system.slice/systemd-journald.service
           └─396 /usr/lib/systemd/systemd-journald

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


[root@localhost:~]$ ls -l /proc/$(pidof systemd-journald)/fd/
insgesamt 0
lr-x------ 1 root root 64 2019-01-14 22:00 0 -> /dev/null
l-wx------ 1 root root 64 2019-01-14 22:00 1 -> /dev/null
lrwx------ 1 root root 64 2019-01-15 01:06 10 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 11 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 12 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 13 -> 'socket:[18692]'
lrwx------ 1 root root 64 2019-01-15 01:06 14 -> 'anon_inode:[timerfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 16 -> 'socket:[15164]'
lrwx------ 1 root root 64 2019-01-14 22:00 17 -> 'socket:[18712]'
lrwx------ 1 root root 64 2019-01-15 01:06 18 -> 'socket:[18714]'
lrwx------ 1 root root 64 2019-01-15 01:06 19 -> 'socket:[18748]'
l-wx------ 1 root root 64 2019-01-14 22:00 2 -> /dev/null
lrwx------ 1 root root 64 2019-01-15 01:06 20 -> 'socket:[15166]'
lrwx------ 1 root root 64 2019-01-15 01:06 21 -> 'socket:[16199]'
lrwx------ 1 root root 64 2019-01-14 22:00 22 -> 'socket:[15170]'
lrwx------ 1 root root 64 2019-01-15 01:06 23 -> 'socket:[13995]'
lrwx------ 1 root root 64 2019-01-14 22:00 24 -> 'socket:[15174]'
lrwx------ 1 root root 64 2019-01-14 22:00 25 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system@e3ed4ab1bd804b0ba69037517d718a71-0000000000008ada-00057f7ced0c66d1.journal
lrwx------ 1 root root 64 2019-01-15 01:06 26 -> 'socket:[15238]'
lrwx------ 1 root root 64 2019-01-15 01:06 27 -> 'socket:[15242]'
lrwx------ 1 root root 64 2019-01-15 01:06 28 -> 'socket:[19921]'
lrwx------ 1 root root 64 2019-01-15 01:06 29 -> 'socket:[19923]'
lrwx------ 1 root root 64 2019-01-14 22:00 3 -> 'socket:[19484]'
lrwx------ 1 root root 64 2019-01-15 01:06 30 -> 'socket:[19925]'
lrwx------ 1 root root 64 2019-01-15 01:06 31 -> 'socket:[24764]'
lrwx------ 1 root root 64 2019-01-15 01:06 32 -> 'socket:[20736]'
lrwx------ 1 root root 64 2019-01-15 01:06 33 -> 'socket:[20618]'
lrwx------ 1 root root 64 2019-01-14 22:00 34 -> 'socket:[21706]'
lrwx------ 1 root root 64 2019-01-15 01:06 35 -> 'socket:[16270]'
lrwx------ 1 root root 64 2019-01-14 22:00 36 -> 'socket:[16272]'
lrwx------ 1 root root 64 2019-01-14 22:00 37 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system.journal
lrwx------ 1 root root 64 2019-01-15 01:06 4 -> 'socket:[19487]'
lrwx------ 1 root root 64 2019-01-15 01:06 5 -> 'socket:[19474]'
l-wx------ 1 root root 64 2019-01-15 01:06 6 -> /dev/kmsg
lrwx------ 1 root root 64 2019-01-15 01:06 7 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 2019-01-15 01:06 8 -> /dev/kmsg
lr-x------ 1 root root 64 2019-01-15 01:06 9 -> /proc/sys/kernel/hostname


[root@localhost:~]$ pmap --extended $(pidof systemd-journald)
396:   /usr/lib/systemd/systemd-journald
Address           Kbytes     RSS   Dirty Mode  Mapping
0000556c15ccd000     132     132       0 r-x-- systemd-journald
0000556c15eed000       8       8       8 r---- systemd-journald
0000556c15eef000       4       4       4 rw--- systemd-journald
0000556c1783f000 14046012   60088   60088 rw---   [ anon ]
00007f6dd2c82000       4       0       0 -----   [ anon ]
00007f6dd2c83000    8192       8       8 rw---   [ anon ]
00007f6dd3483000       4       0       0 -----   [ anon ]
00007f6dd3484000   12296      16      16 rw---   [ anon ]
00007f6dd4689000    6156      12      12 rw---   [ anon ]
00007f6dd4e8d000   12312      24      24 rw---   [ anon ]
00007f6dd5b54000    1280     292     292 rw-s- system.journal
00007f6dd5c94000    8208      16      16 rw---   [ anon ]
00007f6dd6699000    2052       4       4 rw---   [ anon ]
00007f6dd689a000    1608     372       0 r-x-- libm-2.27.so
00007f6dd6a2c000    2048       0       0 ----- libm-2.27.so
00007f6dd6c2c000       4       4       4 r---- libm-2.27.so
00007f6dd6c2d000       4       4       4 rw--- libm-2.27.so
00007f6dd6c2e000     144      64       0 r-x-- libudev.so.1.6.10
00007f6dd6c52000    2048       0       0 ----- libudev.so.1.6.10
00007f6dd6e52000       4       4       4 r---- libudev.so.1.6.10
00007f6dd6e53000       4       4       4 rw--- libudev.so.1.6.10
00007f6dd6e54000     692      64       0 r-x-- libsepol.so.1
00007f6dd6f01000    2048       0       0 ----- libsepol.so.1
00007f6dd7101000       4       4       4 r---- libsepol.so.1
00007f6dd7102000       4       4       4 rw--- libsepol.so.1
00007f6dd7103000       8       0       0 rw---   [ anon ]
00007f6dd7105000    1524     104       0 r-x-- libunistring.so.2.1.0
00007f6dd7282000    2044       0       0 ----- libunistring.so.2.1.0
00007f6dd7481000      16      16      16 r---- libunistring.so.2.1.0
00007f6dd7485000       4       4       4 rw--- libunistring.so.2.1.0
00007f6dd7486000     268      64       0 r-x-- libpcap.so.1.9.0
00007f6dd74c9000    2048       0       0 ----- libpcap.so.1.9.0
00007f6dd76c9000       8       8       8 r---- libpcap.so.1.9.0
00007f6dd76cb000       4       4       4 rw--- libpcap.so.1.9.0
00007f6dd76cc000      56      56       0 r-x-- libjson-c.so.4.0.0
00007f6dd76da000    2044       0       0 ----- libjson-c.so.4.0.0
00007f6dd78d9000       4       4       4 r---- libjson-c.so.4.0.0
00007f6dd78da000       4       4       4 rw--- libjson-c.so.4.0.0
00007f6dd78db000      28      24       0 r-x-- libargon2.so.0
00007f6dd78e2000    2044       0       0 ----- libargon2.so.0
00007f6dd7ae1000       4       4       4 r---- libargon2.so.0
00007f6dd7ae2000       4       4       4 rw--- libargon2.so.0
00007f6dd7ae3000     128     116       0 r-x-- libgpg-error.so.0.25.0
00007f6dd7b03000    2044       0       0 ----- libgpg-error.so.0.25.0
00007f6dd7d02000       4       4       4 r---- libgpg-error.so.0.25.0
00007f6dd7d03000       4       4       4 rw--- libgpg-error.so.0.25.0
00007f6dd7d04000     332     108       0 r-x-- libdevmapper.so.1.02
00007f6dd7d57000    2044       0       0 ----- libdevmapper.so.1.02
00007f6dd7f56000       4       4       4 r---- libdevmapper.so.1.02
00007f6dd7f57000      12      12       8 rw--- libdevmapper.so.1.02
00007f6dd7f5a000       4       0       0 rw---   [ anon ]
00007f6dd7f5b000      24      20       0 r-x-- libuuid.so.1.3.0
00007f6dd7f61000    2044       0       0 ----- libuuid.so.1.3.0
00007f6dd8160000       4       4       4 r---- libuuid.so.1.3.0
00007f6dd8161000       4       0       0 rw---   [ anon ]
00007f6dd8162000      16      16       0 r-x-- libattr.so.1.1.2448
00007f6dd8166000    2048       0       0 ----- libattr.so.1.1.2448
00007f6dd8366000       4       4       4 r---- libattr.so.1.1.2448
00007f6dd8367000       4       0       0 rw---   [ anon ]
00007f6dd8368000      12      12       0 r-x-- libdl-2.27.so
00007f6dd836b000    2044       0       0 ----- libdl-2.27.so
00007f6dd856a000       4       4       4 r---- libdl-2.27.so
00007f6dd856b000       4       4       4 rw--- libdl-2.27.so
00007f6dd856c000     524      60       0 r-x-- libpcre2-8.so.0.7.1
00007f6dd85ef000    2044       0       0 ----- libpcre2-8.so.0.7.1
00007f6dd87ee000       4       4       4 r---- libpcre2-8.so.0.7.1
00007f6dd87ef000       4       4       4 rw--- libpcre2-8.so.0.7.1
00007f6dd87f0000     344      64       0 r-x-- libmount.so.1.1.0
00007f6dd8846000    2044       0       0 ----- libmount.so.1.1.0
00007f6dd8a45000      12      12      12 r---- libmount.so.1.1.0
00007f6dd8a48000       4       4       4 rw--- libmount.so.1.1.0
00007f6dd8a49000       4       0       0 rw---   [ anon ]
00007f6dd8a4a000     300      64       0 r-x-- libblkid.so.1.1.0
00007f6dd8a95000    2044       0       0 ----- libblkid.so.1.1.0
00007f6dd8c94000      20      20      20 r---- libblkid.so.1.1.0
00007f6dd8c99000       4       4       4 rw--- libblkid.so.1.1.0
00007f6dd8c9a000       4       0       0 rw---   [ anon ]
00007f6dd8c9b000      88      60       0 r-x-- liblz4.so.1.8.1
00007f6dd8cb1000    2044       0       0 ----- liblz4.so.1.8.1
00007f6dd8eb0000       4       4       4 r---- liblz4.so.1.8.1
00007f6dd8eb1000       4       0       0 rw---   [ anon ]
00007f6dd8eb2000     148      60       0 r-x-- liblzma.so.5.2.4
00007f6dd8ed7000    2048       0       0 ----- liblzma.so.5.2.4
00007f6dd90d7000       4       4       4 r---- liblzma.so.5.2.4
00007f6dd90d8000       4       0       0 rw---   [ anon ]
00007f6dd90d9000     112      44       0 r-x-- libidn2.so.0.3.4
00007f6dd90f5000    2044       0       0 ----- libidn2.so.0.3.4
00007f6dd92f4000       4       4       4 r---- libidn2.so.0.3.4
00007f6dd92f5000       4       0       0 rw---   [ anon ]
00007f6dd92f6000     176     168       0 r-x-- libseccomp.so.2.3.3
00007f6dd9322000    2048       0       0 ----- libseccomp.so.2.3.3
00007f6dd9522000      84      84      84 r---- libseccomp.so.2.3.3
00007f6dd9537000       4       4       4 rw--- libseccomp.so.2.3.3
00007f6dd9538000      24      24       0 r-x-- libip4tc.so.0.1.0
00007f6dd953e000    2048       0       0 ----- libip4tc.so.0.1.0
00007f6dd973e000       4       4       4 r---- libip4tc.so.0.1.0
00007f6dd973f000       4       4       4 rw--- libip4tc.so.0.1.0
00007f6dd9740000    1108     188       0 r-x-- libgcrypt.so.20.2.4
00007f6dd9855000    2044       0       0 ----- libgcrypt.so.20.2.4
00007f6dd9a54000       8       8       8 r---- libgcrypt.so.20.2.4
00007f6dd9a56000      20      20      20 rw--- libgcrypt.so.20.2.4
00007f6dd9a5b000     320      64       0 r-x-- libcryptsetup.so.12.3.0
00007f6dd9aab000    2048       0       0 ----- libcryptsetup.so.12.3.0
00007f6dd9cab000       4       4       4 r---- libcryptsetup.so.12.3.0
00007f6dd9cac000       8       8       8 rw--- libcryptsetup.so.12.3.0
00007f6dd9cae000      28      28       0 r-x-- libacl.so.1.1.2253
00007f6dd9cb5000    2048       0       0 ----- libacl.so.1.1.2253
00007f6dd9eb5000       4       4       4 r---- libacl.so.1.1.2253
00007f6dd9eb6000       4       4       4 rw--- libacl.so.1.1.2253
00007f6dd9eb7000      16      16       0 r-x-- libcap.so.2.25
00007f6dd9ebb000    2044       0       0 ----- libcap.so.2.25
00007f6dda0ba000       4       4       4 r---- libcap.so.2.25
00007f6dda0bb000       4       4       4 rw--- libcap.so.2.25
00007f6dda0bc000      28      28       0 r-x-- librt-2.27.so
00007f6dda0c3000    2044       0       0 ----- librt-2.27.so
00007f6dda2c2000       4       4       4 r---- librt-2.27.so
00007f6dda2c3000       4       4       4 rw--- librt-2.27.so
00007f6dda2c4000    1748    1296       0 r-x-- libc-2.27.so
00007f6dda479000    2048       0       0 ----- libc-2.27.so
00007f6dda679000      16      16      16 r---- libc-2.27.so
00007f6dda67d000       8       8       8 rw--- libc-2.27.so
00007f6dda67f000      16      12      12 rw---   [ anon ]
00007f6dda683000     100     100       0 r-x-- libpthread-2.27.so
00007f6dda69c000    2048       0       0 ----- libpthread-2.27.so
00007f6dda89c000       4       4       4 r---- libpthread-2.27.so
00007f6dda89d000       4       4       4 rw--- libpthread-2.27.so
00007f6dda89e000      16       4       4 rw---   [ anon ]
00007f6dda8a2000      92      88       0 r-x-- libgcc_s-8-20181215.so.1
00007f6dda8b9000    2044       0       0 ----- libgcc_s-8-20181215.so.1
00007f6ddaab8000       4       4       4 r---- libgcc_s-8-20181215.so.1
00007f6ddaab9000       4       4       4 rw--- libgcc_s-8-20181215.so.1
00007f6ddaaba000     152     128       0 r-x-- libselinux.so.1
00007f6ddaae0000    2044       0       0 ----- libselinux.so.1
00007f6ddacdf000       4       4       4 r---- libselinux.so.1
00007f6ddace0000       4       4       4 rw--- libselinux.so.1
00007f6ddace1000       8       4       4 rw---   [ anon ]
00007f6ddace3000    1904    1448       0 r-x-- libsystemd-shared-238.so
00007f6ddaebf000    2044       0       0 ----- libsystemd-shared-238.so
00007f6ddb0be000     564      64      64 r---- libsystemd-shared-238.so
00007f6ddb14b000       4       4       4 rw--- libsystemd-shared-238.so
00007f6ddb14c000       8       8       8 rw---   [ anon ]
00007f6ddb14e000     156     156       0 r-x-- ld-2.27.so
00007f6ddb21b000    1280     620     620 rw-s- system@e3ed4ab1bd804b0ba69037517d718a71-0000000000008ada-00057f7ced0c66d1.journal
00007f6ddb35b000      60      56      56 rw---   [ anon ]
00007f6ddb36e000       4       4       4 rw-s- system.journal
00007f6ddb370000       4       4       4 rw-s- system@e3ed4ab1bd804b0ba69037517d718a71-0000000000008ada-00057f7ced0c66d1.journal
00007f6ddb371000       4       4       4 rw-s- kernel-seqnum
00007f6ddb372000       8       8       8 rw---   [ anon ]
00007f6ddb374000       4       4       4 r---- ld-2.27.so
00007f6ddb375000       4       4       4 rw--- ld-2.27.so
00007f6ddb376000       4       4       4 rw---   [ anon ]
00007ffe18fb5000     132      28      28 rw---   [ stack ]
00007ffe18fef000      12       0       0 r----   [ anon ]
00007ffe18ff2000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- ------- 
total kB         14172792   66924   61680


[root@localhost:~]$ journalctl --rotate


[root@localhost:~]$ ls -l /proc/$(pidof systemd-journald)/fd/
insgesamt 0
lr-x------ 1 root root 64 2019-01-14 22:00 0 -> /dev/null
l-wx------ 1 root root 64 2019-01-14 22:00 1 -> /dev/null
lrwx------ 1 root root 64 2019-01-15 01:06 10 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 11 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 12 -> 'anon_inode:[signalfd]'
lrwx------ 1 root root 64 2019-01-15 01:06 13 -> 'socket:[18692]'
lrwx------ 1 root root 64 2019-01-15 01:06 14 -> 'anon_inode:[timerfd]'
lrwx------ 1 root root 64 2019-01-14 22:00 15 -> /run/log/journal/d6c85f9dac4f8095c1b641530008165d/system.journal
lrwx------ 1 root root 64 2019-01-15 01:06 16 -> 'socket:[15164]'
lrwx------ 1 root root 64 2019-01-14 22:00 17 -> 'socket:[18712]'
lrwx------ 1 root root 64 2019-01-15 01:06 18 -> 'socket:[18714]'
lrwx------ 1 root root 64 2019-01-15 01:06 19 -> 'socket:[18748]'
l-wx------ 1 root root 64 2019-01-14 22:00 2 -> /dev/null
lrwx------ 1 root root 64 2019-01-15 01:06 20 -> 'socket:[15166]'
lrwx------ 1 root root 64 2019-01-15 01:06 21 -> 'socket:[16199]'
lrwx------ 1 root root 64 2019-01-14 22:00 22 -> 'socket:[15170]'
lrwx------ 1 root root 64 2019-01-15 01:06 23 -> 'socket:[13995]'
lrwx------ 1 root root 64 2019-01-14 22:00 24 -> 'socket:[15174]'
lrwx------ 1 root root 64 2019-01-15 01:06 26 -> 'socket:[15238]'
lrwx------ 1 root root 64 2019-01-15 01:06 27 -> 'socket:[15242]'
lrwx------ 1 root root 64 2019-01-15 01:06 28 -> 'socket:[19921]'
lrwx------ 1 root root 64 2019-01-15 01:06 29 -> 'socket:[19923]'
lrwx------ 1 root root 64 2019-01-14 22:00 3 -> 'socket:[19484]'
lrwx------ 1 root root 64 2019-01-15 01:06 30 -> 'socket:[19925]'
lrwx------ 1 root root 64 2019-01-15 01:06 31 -> 'socket:[24764]'
lrwx------ 1 root root 64 2019-01-15 01:06 32 -> 'socket:[20736]'
lrwx------ 1 root root 64 2019-01-15 01:06 33 -> 'socket:[20618]'
lrwx------ 1 root root 64 2019-01-14 22:00 34 -> 'socket:[21706]'
lrwx------ 1 root root 64 2019-01-15 01:06 35 -> 'socket:[16270]'
lrwx------ 1 root root 64 2019-01-14 22:00 36 -> 'socket:[16272]'
lrwx------ 1 root root 64 2019-01-15 01:06 4 -> 'socket:[19487]'
lrwx------ 1 root root 64 2019-01-15 01:06 5 -> 'socket:[19474]'
l-wx------ 1 root root 64 2019-01-15 01:06 6 -> /dev/kmsg
lrwx------ 1 root root 64 2019-01-15 01:06 7 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 2019-01-15 01:06 8 -> /dev/kmsg
lr-x------ 1 root root 64 2019-01-15 01:06 9 -> /proc/sys/kernel/hostname


[root@localhost:~]$ pmap --extended $(pidof systemd-journald)
396:   /usr/lib/systemd/systemd-journald
Address           Kbytes     RSS   Dirty Mode  Mapping
0000556c15ccd000     132     132       0 r-x-- systemd-journald
0000556c15eed000       8       8       8 r---- systemd-journald
0000556c15eef000       4       4       4 rw--- systemd-journald
0000556c1783f000 14050108   60108   60108 rw---   [ anon ]
00007f6dd2c82000       4       0       0 -----   [ anon ]
00007f6dd2c83000    8192       8       8 rw---   [ anon ]
00007f6dd3483000       4       0       0 -----   [ anon ]
00007f6dd3484000   12296      16      16 rw---   [ anon ]
00007f6dd4689000    6156      12      12 rw---   [ anon ]
00007f6dd4e8d000   12312      24      24 rw---   [ anon ]
00007f6dd5c94000    8208      16      16 rw---   [ anon ]
00007f6dd6559000    1280      56      56 rw-s- system.journal
00007f6dd6699000    2052       4       4 rw---   [ anon ]
00007f6dd689a000    1608     372       0 r-x-- libm-2.27.so
00007f6dd6a2c000    2048       0       0 ----- libm-2.27.so
00007f6dd6c2c000       4       4       4 r---- libm-2.27.so
00007f6dd6c2d000       4       4       4 rw--- libm-2.27.so
00007f6dd6c2e000     144      64       0 r-x-- libudev.so.1.6.10
00007f6dd6c52000    2048       0       0 ----- libudev.so.1.6.10
00007f6dd6e52000       4       4       4 r---- libudev.so.1.6.10
00007f6dd6e53000       4       4       4 rw--- libudev.so.1.6.10
00007f6dd6e54000     692      64       0 r-x-- libsepol.so.1
00007f6dd6f01000    2048       0       0 ----- libsepol.so.1
00007f6dd7101000       4       4       4 r---- libsepol.so.1
00007f6dd7102000       4       4       4 rw--- libsepol.so.1
00007f6dd7103000       8       0       0 rw---   [ anon ]
00007f6dd7105000    1524     104       0 r-x-- libunistring.so.2.1.0
00007f6dd7282000    2044       0       0 ----- libunistring.so.2.1.0
00007f6dd7481000      16      16      16 r---- libunistring.so.2.1.0
00007f6dd7485000       4       4       4 rw--- libunistring.so.2.1.0
00007f6dd7486000     268      64       0 r-x-- libpcap.so.1.9.0
00007f6dd74c9000    2048       0       0 ----- libpcap.so.1.9.0
00007f6dd76c9000       8       8       8 r---- libpcap.so.1.9.0
00007f6dd76cb000       4       4       4 rw--- libpcap.so.1.9.0
00007f6dd76cc000      56      56       0 r-x-- libjson-c.so.4.0.0
00007f6dd76da000    2044       0       0 ----- libjson-c.so.4.0.0
00007f6dd78d9000       4       4       4 r---- libjson-c.so.4.0.0
00007f6dd78da000       4       4       4 rw--- libjson-c.so.4.0.0
00007f6dd78db000      28      24       0 r-x-- libargon2.so.0
00007f6dd78e2000    2044       0       0 ----- libargon2.so.0
00007f6dd7ae1000       4       4       4 r---- libargon2.so.0
00007f6dd7ae2000       4       4       4 rw--- libargon2.so.0
00007f6dd7ae3000     128     116       0 r-x-- libgpg-error.so.0.25.0
00007f6dd7b03000    2044       0       0 ----- libgpg-error.so.0.25.0
00007f6dd7d02000       4       4       4 r---- libgpg-error.so.0.25.0
00007f6dd7d03000       4       4       4 rw--- libgpg-error.so.0.25.0
00007f6dd7d04000     332     108       0 r-x-- libdevmapper.so.1.02
00007f6dd7d57000    2044       0       0 ----- libdevmapper.so.1.02
00007f6dd7f56000       4       4       4 r---- libdevmapper.so.1.02
00007f6dd7f57000      12      12       8 rw--- libdevmapper.so.1.02
00007f6dd7f5a000       4       0       0 rw---   [ anon ]
00007f6dd7f5b000      24      20       0 r-x-- libuuid.so.1.3.0
00007f6dd7f61000    2044       0       0 ----- libuuid.so.1.3.0
00007f6dd8160000       4       4       4 r---- libuuid.so.1.3.0
00007f6dd8161000       4       0       0 rw---   [ anon ]
00007f6dd8162000      16      16       0 r-x-- libattr.so.1.1.2448
00007f6dd8166000    2048       0       0 ----- libattr.so.1.1.2448
00007f6dd8366000       4       4       4 r---- libattr.so.1.1.2448
00007f6dd8367000       4       0       0 rw---   [ anon ]
00007f6dd8368000      12      12       0 r-x-- libdl-2.27.so
00007f6dd836b000    2044       0       0 ----- libdl-2.27.so
00007f6dd856a000       4       4       4 r---- libdl-2.27.so
00007f6dd856b000       4       4       4 rw--- libdl-2.27.so
00007f6dd856c000     524      60       0 r-x-- libpcre2-8.so.0.7.1
00007f6dd85ef000    2044       0       0 ----- libpcre2-8.so.0.7.1
00007f6dd87ee000       4       4       4 r---- libpcre2-8.so.0.7.1
00007f6dd87ef000       4       4       4 rw--- libpcre2-8.so.0.7.1
00007f6dd87f0000     344      64       0 r-x-- libmount.so.1.1.0
00007f6dd8846000    2044       0       0 ----- libmount.so.1.1.0
00007f6dd8a45000      12      12      12 r---- libmount.so.1.1.0
00007f6dd8a48000       4       4       4 rw--- libmount.so.1.1.0
00007f6dd8a49000       4       0       0 rw---   [ anon ]
00007f6dd8a4a000     300      64       0 r-x-- libblkid.so.1.1.0
00007f6dd8a95000    2044       0       0 ----- libblkid.so.1.1.0
00007f6dd8c94000      20      20      20 r---- libblkid.so.1.1.0
00007f6dd8c99000       4       4       4 rw--- libblkid.so.1.1.0
00007f6dd8c9a000       4       0       0 rw---   [ anon ]
00007f6dd8c9b000      88      60       0 r-x-- liblz4.so.1.8.1
00007f6dd8cb1000    2044       0       0 ----- liblz4.so.1.8.1
00007f6dd8eb0000       4       4       4 r---- liblz4.so.1.8.1
00007f6dd8eb1000       4       0       0 rw---   [ anon ]
00007f6dd8eb2000     148      60       0 r-x-- liblzma.so.5.2.4
00007f6dd8ed7000    2048       0       0 ----- liblzma.so.5.2.4
00007f6dd90d7000       4       4       4 r---- liblzma.so.5.2.4
00007f6dd90d8000       4       0       0 rw---   [ anon ]
00007f6dd90d9000     112      44       0 r-x-- libidn2.so.0.3.4
00007f6dd90f5000    2044       0       0 ----- libidn2.so.0.3.4
00007f6dd92f4000       4       4       4 r---- libidn2.so.0.3.4
00007f6dd92f5000       4       0       0 rw---   [ anon ]
00007f6dd92f6000     176     168       0 r-x-- libseccomp.so.2.3.3
00007f6dd9322000    2048       0       0 ----- libseccomp.so.2.3.3
00007f6dd9522000      84      84      84 r---- libseccomp.so.2.3.3
00007f6dd9537000       4       4       4 rw--- libseccomp.so.2.3.3
00007f6dd9538000      24      24       0 r-x-- libip4tc.so.0.1.0
00007f6dd953e000    2048       0       0 ----- libip4tc.so.0.1.0
00007f6dd973e000       4       4       4 r---- libip4tc.so.0.1.0
00007f6dd973f000       4       4       4 rw--- libip4tc.so.0.1.0
00007f6dd9740000    1108     188       0 r-x-- libgcrypt.so.20.2.4
00007f6dd9855000    2044       0       0 ----- libgcrypt.so.20.2.4
00007f6dd9a54000       8       8       8 r---- libgcrypt.so.20.2.4
00007f6dd9a56000      20      20      20 rw--- libgcrypt.so.20.2.4
00007f6dd9a5b000     320      64       0 r-x-- libcryptsetup.so.12.3.0
00007f6dd9aab000    2048       0       0 ----- libcryptsetup.so.12.3.0
00007f6dd9cab000       4       4       4 r---- libcryptsetup.so.12.3.0
00007f6dd9cac000       8       8       8 rw--- libcryptsetup.so.12.3.0
00007f6dd9cae000      28      28       0 r-x-- libacl.so.1.1.2253
00007f6dd9cb5000    2048       0       0 ----- libacl.so.1.1.2253
00007f6dd9eb5000       4       4       4 r---- libacl.so.1.1.2253
00007f6dd9eb6000       4       4       4 rw--- libacl.so.1.1.2253
00007f6dd9eb7000      16      16       0 r-x-- libcap.so.2.25
00007f6dd9ebb000    2044       0       0 ----- libcap.so.2.25
00007f6dda0ba000       4       4       4 r---- libcap.so.2.25
00007f6dda0bb000       4       4       4 rw--- libcap.so.2.25
00007f6dda0bc000      28      28       0 r-x-- librt-2.27.so
00007f6dda0c3000    2044       0       0 ----- librt-2.27.so
00007f6dda2c2000       4       4       4 r---- librt-2.27.so
00007f6dda2c3000       4       4       4 rw--- librt-2.27.so
00007f6dda2c4000    1748    1296       0 r-x-- libc-2.27.so
00007f6dda479000    2048       0       0 ----- libc-2.27.so
00007f6dda679000      16      16      16 r---- libc-2.27.so
00007f6dda67d000       8       8       8 rw--- libc-2.27.so
00007f6dda67f000      16      12      12 rw---   [ anon ]
00007f6dda683000     100     100       0 r-x-- libpthread-2.27.so
00007f6dda69c000    2048       0       0 ----- libpthread-2.27.so
00007f6dda89c000       4       4       4 r---- libpthread-2.27.so
00007f6dda89d000       4       4       4 rw--- libpthread-2.27.so
00007f6dda89e000      16       4       4 rw---   [ anon ]
00007f6dda8a2000      92      88       0 r-x-- libgcc_s-8-20181215.so.1
00007f6dda8b9000    2044       0       0 ----- libgcc_s-8-20181215.so.1
00007f6ddaab8000       4       4       4 r---- libgcc_s-8-20181215.so.1
00007f6ddaab9000       4       4       4 rw--- libgcc_s-8-20181215.so.1
00007f6ddaaba000     152     128       0 r-x-- libselinux.so.1
00007f6ddaae0000    2044       0       0 ----- libselinux.so.1
00007f6ddacdf000       4       4       4 r---- libselinux.so.1
00007f6ddace0000       4       4       4 rw--- libselinux.so.1
00007f6ddace1000       8       4       4 rw---   [ anon ]
00007f6ddace3000    1904    1448       0 r-x-- libsystemd-shared-238.so
00007f6ddaebf000    2044       0       0 ----- libsystemd-shared-238.so
00007f6ddb0be000     564      64      64 r---- libsystemd-shared-238.so
00007f6ddb14b000       4       4       4 rw--- libsystemd-shared-238.so
00007f6ddb14c000       8       8       8 rw---   [ anon ]
00007f6ddb14e000     156     156       0 r-x-- ld-2.27.so
00007f6ddb35b000      60      56      56 rw---   [ anon ]
00007f6ddb36f000       4       4       4 rw-s- system.journal
00007f6ddb371000       4       4       4 rw-s- kernel-seqnum
00007f6ddb372000       8       8       8 rw---   [ anon ]
00007f6ddb374000       4       4       4 r---- ld-2.27.so
00007f6ddb375000       4       4       4 rw--- ld-2.27.so
00007f6ddb376000       4       4       4 rw---   [ anon ]
00007ffe18fb5000     132      28      28 rw---   [ stack ]
00007ffe18fef000      12       0       0 r----   [ anon ]
00007ffe18ff2000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         14175604   66084   60840


[root@localhost:~]$ systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
  Drop-In: /etc/systemd/system/systemd-journald.service.d
           └─hidepid.conf, security.conf
   Active: active (running) since Mon 2019-01-14 22:00:33 CET; 14h ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 396 (systemd-journal)
   Status: "Processing requests..."
    Tasks: 1 (limit: 512)
   Memory: 101.2M
   CGroup: /system.slice/systemd-journald.service
           └─396 /usr/lib/systemd/systemd-journald

Comment 4 Harald Reindl 2019-01-15 11:15:33 UTC
Created attachment 1520721 [details]
dead symlinks in ls -l /proc/$(pidof systemd-journald)/fd/

ls -l /proc/$(pidof systemd-journald)/fd/ shows all the symlinks to "socket" and "anon" are broken

Comment 5 Harald Reindl 2019-01-15 16:56:43 UTC
-------------------------------------------------------------------------
Debian Security Advisory DSA-4367-2 security@debian.org
https://www.debian.org/security/ Salvatore Bonaccorso
January 15, 2019 https://www.debian.org/security/faq
-------------------------------------------------------------------------
Package : systemd
The Qualys Research Labs reported that the backported security fixes
shipped in DSA 4367-1 contained a memory leak in systemd-journald. This
and an unrelated bug in systemd-coredump are corrected in this update.
Note that as the systemd-journald service is not restarted automatically
a restart of the service or more safely a reboot is advised.
For the stable distribution (stretch), these problems have been fixed in
version 232-25+deb9u8.

Comment 6 Zbigniew Jędrzejewski-Szmek 2019-01-15 22:22:00 UTC
> ls -l /proc/$(pidof systemd-journald)/fd/ shows all the symlinks to "socket" and "anon" are broken
That's expected, those are not real files. Those symlinks are always "broken".

> The Qualys Research Labs reported that the backported security fixes
> shipped in DSA 4367-1 contained a memory leak in systemd-journald. This
> and an unrelated bug in systemd-coredump are corrected in this update.
I haven't had any reports that there was a bug in the upstream patch. This
text implies that the text was in the backport to systemd-232. Do you have the
link to the change?

Is this problem a regression you're observing with systemd-238-11.gita76ee90.fc28?
I looked over the commits that were added again in -11, and I don't see any leak.

Comment 7 Harald Reindl 2019-01-15 22:44:17 UTC
at least i can't remember memory growing that fast while it was too fat anyways, frankly when wahtever software which don't have to write anything to disk and is limited to 10 MB memory by it#s configuration a) grows to VIRTL in the meantime way over 30 GB and c) has a RES of far above 100 MB with gowing numbers even after a rotate (whatever is there to rotate in that setup) somehting is broken

the majority of small VMs hat journald currently on peak of memory usage and there is no valid reason for more that lets say 30 MB in that config which is to be honest also too much, a proper basic OS (sshd, rsyslog, nothing else) needs 70 MB after boot

BTW: what is the point of "always broken symlinks" - don't look like a proper design

Comment 8 samoht0 2019-01-16 18:41:06 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #6)
> I haven't had any reports that there was a bug in the upstream patch. This
> text implies that the text was in the backport to systemd-232. Do you have
> the link to the change?

Here is the systemd stretch git:
https://salsa.debian.org/systemd-team/systemd/commits/stretch

I assume it's d9c31850 from Jan 14, 2019. Hope that helps.

Comment 9 Harald Reindl 2019-01-16 19:02:22 UTC
Created attachment 1521105 [details]
tiny server after 1 day uptime

you can argue what you want, that is a joke on a tiny server running nothing else then asterisk and named while named is auth-only and so no caching and don't run syslog at all

the whole machine could easily live with the amount of memory journald eats

Comment 10 Zbigniew Jędrzejewski-Szmek 2019-01-17 09:01:09 UTC
Thanks. There's also #1666690. Both bugs are the same, but they are not present in the systemd-238 in F28. I tried the reproducer provided, and it doesn't reproduce. So while the memory usage does look suspiciously high, it doesn't seem to be related to recent changes.

Comment 11 Harald Reindl 2019-01-19 18:21:58 UTC
please take also a look at that graph https://github.com/systemd/systemd/issues/9141#issuecomment-455802336 amd maybe it's suspiciously high longer, i am not always looking at journald memorry usage because i expected "RuntimeMaxUse=10M" just to work

anyways, on a VOIP machine with 768 MB RAM and 524 MB used having journald 110 MB RES and 25.4GB VIRT is simply not acceptable and writing cronjobs restart journald every few hours feels so 1990

Comment 12 samoht0 2019-02-02 09:08:36 UTC
This is considered fixed
https://github.com/systemd/systemd/issues/11502
with commit
https://github.com/msekletar/systemd/commit/8bb299ce155abc7cf87718d071b2a1fd9e64760d

Harald, can you possibly test that commit based on F28 source?

Comment 13 Harald Reindl 2019-02-02 12:13:15 UTC
only if somebody provides at least a wokring scratch build

while VIRT is terrible high and needs to be fixed too i care more about 70 MB to 200 MB after a view days uptime which leads to oom-killer on machines just running an authoritative nameserver which is pretty fine with a 768 MB RAM virtual machine and other servcies like dedicated sftp-server only running sshd at large dnf-transactions without forcibly restart journald before

Comment 14 Fedora Update System 2019-02-07 19:56:20 UTC
systemd-239-10.git4dc7dce.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fb1547321

Comment 15 Fedora Update System 2019-02-10 04:28:08 UTC
systemd-239-11.git4dc7dce.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fb1547321

Comment 16 Fedora Update System 2019-02-11 01:57:25 UTC
systemd-239-11.git4dc7dce.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Japheth Cleaver 2019-05-07 22:29:21 UTC
This apparently was only fixed on F29, not F28. Can this be put through before support closes?

Comment 18 Harald Reindl 2019-05-08 13:01:47 UTC
CLOSED ERRATA: it's amazing that nobody cares about the reported fedora release all the time

fact is that journalctl --rotate also don't work at all on F28
it just doubles the stuff below /run/log without cleanup anything


Note You need to log in before you can comment on or make changes to this bug.