Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
No more logging of anything, just the following stuff in /var/log/messages;
restarting of rsyslog or the whole system does not help:
Jul 19 03:42:03 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="615" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jul 21 16:27:22 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="614" x-info="http://www.rsyslog.com"] start
Jul 21 16:27:22 backup rsyslogd-2027: imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
[try http://www.rsyslog.com/e/2027 ]
Jul 21 14:39:35 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="614" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul 21 14:39:35 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1355" x-info="http://www.rsyslog.com"] start
Jul 21 14:39:35 backup rsyslogd-2027: imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
[try http://www.rsyslog.com/e/2027 ]
Version-Release number of selected component (if applicable):
rsyslog-7.4.7-7.el7_0.x86_64
How reproducible:
No idea - but it feels strange.
Actual results:
imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
Expected results:
No errors.
Additional info:
[root@backup log]# cd /var/lib/rsyslog/
[root@backup rsyslog]# ls -l
insgesamt 0
-rw-r--r--. 1 root root 0 1. Sep 2014 imjournal.state
[root@backup rsyslog]#
[root@backup rsyslog]# mv /var/lib/rsyslog/imjournal.state /var/lib/rsyslog/imjournal.state.broken
[root@backup rsyslog]#
[root@backup rsyslog]# systemctl restart rsyslog.service
[root@backup rsyslog]#
Afterwards it seems to have settled again:
Jul 21 14:39:48 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1355" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul 21 14:39:48 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1371" x-info="http://www.rsyslog.com"] start
Jul 21 16:27:12 backup journal: Runtime journal is using 8.0M (max 399.1M, leaving 598.7M of free 3.8G, current limit 399.1M).
Jul 21 16:27:12 backup journal: Runtime journal is using 8.0M (max 399.1M, leaving 598.7M of free 3.8G, current limit 399.1M).
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpuset
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpu
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpuacct
[...]
The question is: Why does /var/lib/rsyslog/imjournal.state hold up the
whole logging? This feels like a bug to me.
Created attachment 1170158[details]
Patch that resolves reading and writing of statefile
This bug is caused by not very atomic writing of imjournal statefile and it's hardly reproducible but there is a way.
fscanf error appears only when rsyslog reads empty statefile and it will cause that imjournal is stopped so no logging from journal is performed.
When statefile contains some random bytes error appears again but from journal and imjournal is stopped too.
In this patch Rsyslog is writing imjournal statefile more atomically and secure.
Reading of statefile is more robust and it doesn't affect imjournal module so when corrupted statefile is read then imjournal ignore statefile and continue with logging and it doesn't stop. We can use a logger as a test if it's logging or not.
Patch introduce new option in both old and new config "IgnoreNonValidStateFile" which is default "on" and it can turn off ignorance of non valid statefile.
New config:
module(load="imjournal"
StateFile="imjournal.state"
IgnoreNonValidStatefile="on"/"off" #default on
#IgnorePreviousMessages="on"
)
Old config:
$ModLoad imjournal
$WorkDirectory /var/lib/rsyslog
$IMJournalStateFile imjournal.state
$IMJournalIgnoreNonValidStatefile on/off
#$IMJournalIgnorePreviousMessages on
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHEA-2016-2401.html