RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1245194 - imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
Summary: imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog
Version: 7.1
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Peter Vrabec
QA Contact: Stefan Dordevic
URL:
Whiteboard:
Depends On:
Blocks: 1203710 1295396 1296594 1313485
TreeView+ depends on / blocked
 
Reported: 2015-07-21 12:46 UTC by Robert Scheck
Modified: 2020-12-11 11:52 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-04 05:36:29 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Patch that resolves reading and writing of statefile (6.20 KB, patch)
2016-06-21 08:50 UTC, Radovan Sroka 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1599223 0 None None None Never
Red Hat Product Errata RHEA-2016:2401 0 normal SHIPPED_LIVE rsyslog bug fix and enhancement update 2016-11-03 13:55:50 UTC

Description Robert Scheck 2015-07-21 12:46:09 UTC 
Description of problem:
No more logging of anything, just the following stuff in /var/log/messages;
restarting of rsyslog or the whole system does not help:

Jul 19 03:42:03 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="615" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jul 21 16:27:22 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="614" x-info="http://www.rsyslog.com"] start
Jul 21 16:27:22 backup rsyslogd-2027: imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
 [try http://www.rsyslog.com/e/2027 ]
Jul 21 14:39:35 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="614" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul 21 14:39:35 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1355" x-info="http://www.rsyslog.com"] start
Jul 21 14:39:35 backup rsyslogd-2027: imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed
 [try http://www.rsyslog.com/e/2027 ]

Version-Release number of selected component (if applicable):
rsyslog-7.4.7-7.el7_0.x86_64

How reproducible:
No idea - but it feels strange.

Actual results:
imjournal: fscanf on state file `/var/lib/rsyslog/imjournal.state' failed

Expected results:
No errors.

Additional info:
[root@backup log]# cd /var/lib/rsyslog/
[root@backup rsyslog]# ls -l
insgesamt 0
-rw-r--r--. 1 root root 0  1. Sep 2014  imjournal.state
[root@backup rsyslog]# 

[root@backup rsyslog]# mv /var/lib/rsyslog/imjournal.state /var/lib/rsyslog/imjournal.state.broken
[root@backup rsyslog]# 

[root@backup rsyslog]# systemctl restart rsyslog.service
[root@backup rsyslog]# 

Afterwards it seems to have settled again:

Jul 21 14:39:48 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1355" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul 21 14:39:48 backup rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="1371" x-info="http://www.rsyslog.com"] start
Jul 21 16:27:12 backup journal: Runtime journal is using 8.0M (max 399.1M, leaving 598.7M of free 3.8G, current limit 399.1M).
Jul 21 16:27:12 backup journal: Runtime journal is using 8.0M (max 399.1M, leaving 598.7M of free 3.8G, current limit 399.1M).
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpuset
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpu
Jul 21 16:27:12 backup kernel: Initializing cgroup subsys cpuacct
[...]

The question is: Why does /var/lib/rsyslog/imjournal.state hold up the
whole logging? This feels like a bug to me.
Comment 2 Robert Scheck 2015-07-21 12:49:00 UTC 
Cross-filed case 01479984 on Red Hat customer portal.
Comment 10 Timothy Murphy 2015-10-23 12:35:39 UTC 
I'm getting this message on my Fedora-23beta/KDE laptop, when I restart rsyslog in order to get information in /var/log/messages .
Comment 16 Radovan Sroka 2016-06-21 08:50:54 UTC 
Created attachment 1170158 [details]
Patch that resolves reading and writing of statefile

This bug is caused by not very atomic writing of imjournal statefile and it's hardly reproducible but there is a way.

fscanf error appears only when rsyslog reads empty statefile and it will cause that imjournal is stopped so no logging from journal is performed.
When statefile contains some random bytes error appears again but from journal and imjournal is stopped too.


In this patch Rsyslog is writing imjournal statefile more atomically and secure.

Reading of statefile is more robust and it doesn't affect imjournal module so when corrupted statefile is read then imjournal ignore statefile and continue with logging and it doesn't stop. We can use a logger as a test if it's logging or not.

Patch introduce new option in both old and new config "IgnoreNonValidStateFile" which is default "on" and it can turn off ignorance of non valid statefile.

New config:

module(load="imjournal" 
    StateFile="imjournal.state" 
    IgnoreNonValidStatefile="on"/"off" #default on
    #IgnorePreviousMessages="on"
)

Old config:

$ModLoad imjournal
$WorkDirectory /var/lib/rsyslog
$IMJournalStateFile imjournal.state
$IMJournalIgnoreNonValidStatefile on/off
#$IMJournalIgnorePreviousMessages on
Comment 26 errata-xmlrpc 2016-11-04 05:36:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2401.html
Comment 27 popolon72 2016-11-23 08:18:41 UTC 
Is this change going to be put to upstream to github.com rsyslog repository?
Comment 28 Radovan Sroka 2016-11-23 11:17:58 UTC 
Yes I hope so,

Most likely in near future.
Comment 29 Radovan Sroka 2016-12-08 16:00:54 UTC 
We are already created a pull request on upstream github repository.

https://github.com/rsyslog/rsyslog/pull/1317
Note You need to log in before you can comment on or make changes to this bug.