Bug 1656860

Summary: rsyslog imfile buffer overflow
Product: Red Hat Enterprise Linux 7 Reporter: Sven <s.buesing>
Component: rsyslogAssignee: Jiří Vymazal <jvymazal>
Status: CLOSED ERRATA QA Contact: Radovan Sroka <rsroka>
Severity: medium Docs Contact:
Priority: urgent    
Version: 7.6CC: bjarolim, dapospis, fkrska, james, jvymazal, lvrabec, pasik, providing, pvrabec, rmeggins, rsroka
Target Milestone: rcKeywords: Regression, Reproducer
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rsyslog-8.24.0-38.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 12:48:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1649250    
Bug Blocks:    
Attachments:
Description Flags
containers.conf - rsyslog.d/ configuration file none

Description Sven 2018-12-06 14:00:56 UTC 
Created attachment 1512143 [details]
containers.conf - rsyslog.d/ configuration file

Description of problem:

We use the attached containers.conf configuration.
The configuration loads the imfile module and reads logfiles below /var/log/containers/*.log via inotify.

If the number of files below /var/log/containers exceeds around 30 Files rsyslog is not able to start up und throws a buffer overflow exception.

The Logfiles below /var/log/containers/*.log are symlinks to the actual logfiles. See Actual Results for Stacktrace.


Version-Release number of selected component (if applicable): rsyslog-8.24.0-34.el7.x86_64


How reproducible: Could not reproduce on a system without workload. Still trying.


Steps to Reproduce:
1.
2.
3.

Actual results: 

/usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS
*** buffer overflow detected ***: /usr/sbin/rsyslogd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f9959e799e7]
/lib64/libc.so.6(+0x115b62)[0x7f9959e77b62]
/usr/lib64/rsyslog/imfile.so(+0x6351)[0x7f995953b351]
/usr/lib64/rsyslog/imfile.so(+0x685e)[0x7f995953b85e]
/usr/lib64/rsyslog/imfile.so(+0x263d)[0x7f995953763d]
/usr/lib64/rsyslog/imfile.so(+0x2649)[0x7f9959537649]
/usr/lib64/rsyslog/imfile.so(+0x2649)[0x7f9959537649]
/usr/lib64/rsyslog/imfile.so(+0x2649)[0x7f9959537649]
/usr/lib64/rsyslog/imfile.so(+0x2649)[0x7f9959537649]
/usr/lib64/rsyslog/imfile.so(+0x5930)[0x7f995953a930]
/usr/lib64/rsyslog/imfile.so(+0x5e9c)[0x7f995953ae9c]
/usr/sbin/rsyslogd(+0x48154)[0x557fa8cda154]
/lib64/libpthread.so.0(+0x7dd5)[0x7f995ad6bdd5]
/lib64/libc.so.6(clone+0x6d)[0x7f9959e5fead]
======= Memory map: ========
557fa8c92000-557fa8d27000 r-xp 00000000 fd:0a 795374                     /usr/sbin/rsyslogd
557fa8f27000-557fa8f2a000 r--p 00095000 fd:0a 795374                     /usr/sbin/rsyslogd
557fa8f2a000-557fa8f31000 rw-p 00098000 fd:0a 795374                     /usr/sbin/rsyslogd
557fa8f31000-557fa8f32000 rw-p 00000000 00:00 0 
557fa9665000-557fa96a8000 rw-p 00000000 00:00 0                          [heap]
7f993c000000-7f993c04f000 rw-p 00000000 00:00 0 
7f993c04f000-7f9940000000 ---p 00000000 00:00 0 
7f9944000000-7f9946533000 rw-p 00000000 00:00 0 
7f9946533000-7f9948000000 ---p 00000000 00:00 0 
7f994b5e6000-7f994b5e7000 ---p 00000000 00:00 0 
7f994b5e7000-7f994bde7000 rw-p 00000000 00:00 0 
7f994bde7000-7f994bdfd000 r-xp 00000000 fd:0a 790168                     /usr/lib64/libresolv-2.17.so
7f994bdfd000-7f994bffc000 ---p 00016000 fd:0a 790168                     /usr/lib64/libresolv-2.17.so
7f994bffc000-7f994bffd000 r--p 00015000 fd:0a 790168                     /usr/lib64/libresolv-2.17.so
7f994bffd000-7f994bffe000 rw-p 00016000 fd:0a 790168                     /usr/lib64/libresolv-2.17.so
7f994bffe000-7f994c000000 rw-p 00000000 00:00 0 
7f994c000000-7f9950000000 rw-p 00000000 00:00 0 
7f9950000000-7f995006c000 rw-p 00000000 00:00 0 
7f995006c000-7f9954000000 ---p 00000000 00:00 0 
7f9954000000-7f9954021000 rw-p 00000000 00:00 0 
7f9954021000-7f9958000000 ---p 00000000 00:00 0 
7f9958129000-7f995812e000 r-xp 00000000 fd:0a 790156                     /usr/lib64/libnss_dns-2.17.so
7f995812e000-7f995832e000 ---p 00005000 fd:0a 790156                     /usr/lib64/libnss_dns-2.17.so
7f995832e000-7f995832f000 r--p 00005000 fd:0a 790156                     /usr/lib64/libnss_dns-2.17.so
7f995832f000-7f9958330000 rw-p 00006000 fd:0a 790156                     /usr/lib64/libnss_dns-2.17.so
7f9958330000-7f9958331000 ---p 00000000 00:00 0 
7f9958331000-7f9958b31000 rw-p 00000000 00:00 0 
7f9958b31000-7f9958b32000 ---p 00000000 00:00 0 
7f9958b32000-7f9958f32000 rw-p 00000000 00:00 0 
7f9958f32000-7f9958f33000 ---p 00000000 00:00 0 
7f9958f33000-7f9959333000 rw-p 00000000 00:00 0 
7f9959333000-7f9959334000 r-xp 00000000 fd:0a 795353                     /usr/lib64/rsyslog/lmregexp.so
7f9959334000-7f9959533000 ---p 00001000 fd:0a 795353                     /usr/lib64/rsyslog/lmregexp.so
7f9959533000-7f9959534000 r--p 00000000 fd:0a 795353                     /usr/lib64/rsyslog/lmregexp.so
7f9959534000-7f9959535000 rw-p 00001000 fd:0a 795353                     /usr/lib64/rsyslog/lmregexp.so
7f9959535000-7f995953f000 r-xp 00000000 fd:0a 795340                     /usr/lib64/rsyslog/imfile.so
7f995953f000-7f995973e000 ---p 0000a000 fd:0a 795340                     /usr/lib64/rsyslog/imfile.so
7f995973e000-7f995973f000 r--p 00009000 fd:0a 795340                     /usr/lib64/rsyslog/imfile.so
7f995973f000-7f9959740000 rw-p 0000a000 fd:0a 795340                     /usr/lib64/rsyslog/imfile.so
7f9959740000-7f9959747000 r-xp 00000000 fd:0a 795349                     /usr/lib64/rsyslog/imuxsock.so
7f9959747000-7f9959947000 ---p 00007000 fd:0a 795349                     /usr/lib64/rsyslog/imuxsock.so
7f9959947000-7f9959948000 r--p 00007000 fd:0a 795349                     /usr/lib64/rsyslog/imuxsock.so
7f9959948000-7f9959949000 rw-p 00008000 fd:0a 795349                     /usr/lib64/rsyslog/imuxsock.so
7f9959949000-7f9959955000 r-xp 00000000 fd:0a 790158                     /usr/lib64/libnss_files-2.17.so
7f9959955000-7f9959b54000 ---p 0000c000 fd:0a 790158                     /usr/lib64/libnss_files-2.17.so
7f9959b54000-7f9959b55000 r--p 0000b000 fd:0a 790158                     /usr/lib64/libnss_files-2.17.so
7f9959b55000-7f9959b56000 rw-p 0000c000 fd:0a 790158                     /usr/lib64/libnss_files-2.17.so
7f9959b56000-7f9959b5c000 rw-p 00000000 00:00 0 
7f9959b5c000-7f9959b61000 r-xp 00000000 fd:0a 803493                     /usr/lib64/rsyslog/lmnet.so
7f9959b61000-7f9959d60000 ---p 00005000 fd:0a 803493                     /usr/lib64/rsyslog/lmnet.so
7f9959d60000-7f9959d61000 r--p 00004000 fd:0a 803493                     /usr/lib64/rsyslog/lmnet.so
7f9959d61000-7f9959d62000 rw-p 00005000 fd:0a 803493                     /usr/lib64/rsyslog/lmnet.so
7f9959d62000-7f9959f24000 r-xp 00000000 fd:0a 790140                     /usr/lib64/libc-2.17.so
7f9959f24000-7f995a124000 ---p 001c2000 fd:0a 790140                     /usr/lib64/libc-2.17.so
7f995a124000-7f995a128000 r--p 001c2000 fd:0a 790140                     /usr/lib64/libc-2.17.so
7f995a128000-7f995a12a000 rw-p 001c6000 fd:0a 790140                     /usr/lib64/libc-2.17.so
7f995a12a000-7f995a12f000 rw-p 00000000 00:00 0 
7f995a12f000-7f995a144000 r-xp 00000000 fd:0a 799944                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f995a144000-7f995a343000 ---p 00015000 fd:0a 799944                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f995a343000-7f995a344000 r--p 00014000 fd:0a 799944                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f995a344000-7f995a345000 rw-p 00015000 fd:0a 799944                     /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f995a345000-7f995a349000 r-xp 00000000 fd:0a 791049                     /usr/lib64/libuuid.so.1.3.0
7f995a349000-7f995a548000 ---p 00004000 fd:0a 791049                     /usr/lib64/libuuid.so.1.3.0
7f995a548000-7f995a549000 r--p 00003000 fd:0a 791049                     /usr/lib64/libuuid.so.1.3.0
7f995a549000-7f995a54a000 rw-p 00004000 fd:0a 791049                     /usr/lib64/libuuid.so.1.3.0
7f995a54a000-7f995a553000 r-xp 00000000 fd:0a 793231                     /usr/lib64/libfastjson.so.4.0.0
7f995a553000-7f995a752000 ---p 00009000 fd:0a 793231                     /usr/lib64/libfastjson.so.4.0.0
7f995a752000-7f995a753000 r--p 00008000 fd:0a 793231                     /usr/lib64/libfastjson.so.4.0.0
7f995a753000-7f995a754000 rw-p 00009000 fd:0a 793231                     /usr/lib64/libfastjson.so.4.0.0
7f995a754000-7f995a757000 r-xp 00000000 fd:0a 793228                     /usr/lib64/libestr.so.0.0.0Aborted

Expected results:
Rsyslog should start without a buffer overflow.


Additional info:
There is a similiar issue on rsyslogs github. https://github.com/rsyslog/rsyslog/issues/133
Comment 2 Sven 2018-12-06 14:10:23 UTC 
Supplement: This error only occured on our systems after upgrading to 7.6 which upgraded rsyslog from version 8.24.0-16 to rsyslog-8.24.0-34
Comment 3 Jiří Vymazal 2018-12-10 14:55:33 UTC 
Hi Sven,

Any updated regarding a reproducer? I tried few things but I was unable to reproduce the issue. Are you able to try if rsyslog crashes when the log files are created while rsyslog is running or only when started with the files already in place? Also, when rsyslog is restarted after the crash does it crash again? If not does the collecting of logs from files work as expected?

Thanks for answers.
Comment 4 Sven 2018-12-21 15:56:54 UTC 
Hi Jiří,

sorry for the delay. It took me a while to find a case where I could reproduce the problem reliably.

I created a vagrant machine so you could easily follow the problem:
https://github.com/svenbs/centos7-rsyslogd-bufferoverflow-reproduction

Regards,
Sven
Comment 5 F. Bernattzki 2019-01-14 16:01:25 UTC 
Is there any timeline for fixing this?
Comment 6 Jiří Vymazal 2019-01-15 08:25:42 UTC 
(In reply to F. Bernattzki from comment #5)
> Is there any timeline for fixing this?

This bug is currently being evaluated regarding inclusion in next release/rsyslog update. The timeline will depend on result of that.
Comment 8 Sven 2019-02-06 09:28:59 UTC 
Hi Jiří,

any news on this? Is there a releasedate for a new version including the fix?

Regards,
Sven
Comment 35 errata-xmlrpc 2019-08-06 12:48:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2110