Bug 974496

Summary: rsyslogd consuming huge amounts of system resources by rapidly writing old log messages
Product: [Fedora] Fedora Reporter: Aaron Sowry <aaron>
Component: rsyslogAssignee: Tomas Heinrich <theinric>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: aaron, mah.darade, pvrabec, theinric
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-14 13:01:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aaron Sowry 2013-06-14 09:41:16 UTC
Description of problem:
I have no clue what's going on here.

rsyslogd is rapidly and repeatedly writing month-old log messages to /var/log/messages, consuming huge amounts of CPU and disk space in the process.

$ sudo systemctl start rsyslog.service
$ pgrep rsyslog
2624
$ sudo strace -fp 2624
Process 2624 attached with 4 threads
[pid  2630] read(3,  <unfinished ...>
[pid  2631] futex(0x7f45803d3844, FUTEX_WAIT_PRIVATE, 479301, NULL <unfinished ...>
[pid  2629] futex(0x7f45803d3844, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f45803d3840, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid  2631] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid  2629] <... futex resumed> )       = 0
[pid  2624] select(1, NULL, NULL, NULL, {86381, 541741} <unfinished ...>
[pid  2631] futex(0x7f45803d3a10, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  2629] futex(0x7f45803d3a10, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid  2631] <... futex resumed> )       = 0
[pid  2629] <... futex resumed> )       = 1
[pid  2631] futex(0x7f45803d3a10, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2631] write(5, "May  9 11:13:41 wotan systemd: M"..., 49) = 49
[pid  2631] futex(0x7f45803d3844, FUTEX_WAIT_PRIVATE, 479303, NULL <unfinished ...>
[pid  2629] futex(0x7f45803d3844, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f45803d3840, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid  2631] <... futex resumed> )       = 0
[pid  2629] <... futex resumed> )       = 1
[pid  2631] futex(0x7f45803d3a10, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  2629] futex(0x7f45803d3a10, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid  2631] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid  2629] <... futex resumed> )       = 0
[pid  2631] futex(0x7f45803d3a10, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  2631] write(5, "May  9 11:13:41 wotan systemd: M"..., 46) = 46
[pid  2631] futex(0x7f45803d3844, FUTEX_WAIT_PRIVATE, 479305, NULL <unfinished ...>
[pid  2629] futex(0x7f45803d3844, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f45803d3840, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid  2631] <... futex resumed> )       = 0
[pid  2629] <... futex resumed> )       = 1
...

You can see that the log messages being written are for May 9th; the strace was run today.

My initial thought was that the rsyslog resource usage was caused by bug #974200, but after applying the fix, I noticed that the logging had nothing to do with this bug. Because of bug #974200 however, my logs became fairly massive - not sure if this could have anything to do with it.

Version-Release number of selected component (if applicable):
rsyslog-7.4.0-1.fc19.x86_64

How reproducible:
Always (at least for me)

Steps to Reproduce:
1. Start rsyslogd
2.
3.

Comment 1 Tomas Heinrich 2013-06-14 13:01:18 UTC

*** This bug has been marked as a duplicate of bug 974132 ***