Red Hat Bugzilla – Bug 983230
systemd-journald leaks buffer_head slab objects
Last modified: 2013-07-12 14:53:27 EDT
Description of problem:
If systemd-journald is running, and some process is doing printk's, buffer_head slab objects increase and never decrease
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Make sure systemd-journald is running (but rsyslog.service can be stopped)
2. Start a process to cause printks. I am using this:
while : ; do modprobe nf_conntrack ; modprobe -r nf_conntrack ; done
which simply prints out the nf_conntrack version on each module load cycle
3. In another window, launch slabtop
4. Note buffer_head count increasing in slabtop
5. Stop systemd-journald service, and note that buffer_head count stops increasing
(In reply to Phil Oester from comment #0)
> If systemd-journald is running, and some process is doing printk's,
> buffer_head slab objects increase and never decrease
Does it have to be printk? Do you not observe the same phenomenon using logs originating in userspace? Such as:
while :; do
for i in `seq 1000`; do
slabtop -o | grep buffer_head
More importantly: Does the slab grow completely out of bounds? Does it eventually exhaust all RAM and cause an Out-Of-Memory condition? I don't think it would do that. It's supposed to be reclaimable when there's memory pressure.
Does it shrink when you do "echo 3 > /proc/sys/vm/drop_caches"?
What filesystem is your /var/log/journal on? ext4?
I think this is NOTABUG, but I am reassigning to kernel first (because if it were a real bug, it would be in the kernel).
I booted the box with less ram (mem=), and it does not OOM.
Was not aware that systemd-journald was writing binary logfile data to disk. I assumed with rsyslogd shutdown, all disk logging activity should be halted (like the "old days"). Now apparently you have to stop two separate logging services to stop disk logging.
So this buffer_head usage appears to be normal caching, and this report can be closed as NOTABUG.
OK, thanks for testing this.
I strongly recommend against disabling systemd-journald. Rather, in /etc/systemd/journald.conf set "Storage=" to "volatile", or "none". See man journald.conf for details.