Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
We have a couple of customers hitting a "buffer overflow" when more than 1024 file descriptors are used.
Backtrace shows this:
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fae68aef577]
/lib64/libc.so.6(+0x1166f2)[0x7fae68aed6f2]
/lib64/libc.so.6(+0x1184d7)[0x7fae68aef4d7]
/usr/lib64/rsyslog/lmnsd_ptcp.so(+0x2ea4)[0x7fae636fdea4]
/usr/lib64/rsyslog/lmtcpsrv.so(+0x419c)[0x7fae64e6719c]
/usr/sbin/rsyslogd(+0x484f4)[0x55cb97eab4f4]
/lib64/libpthread.so.0(+0x7ea5)[0x7fae699e1ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fae68ad58dd]
======= Memory map: ========
...
And gdb backtrace:
(gdb) bt
#0 0x00007ffff69e81d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff69e98c8 in __GI_abort () at abort.c:90
#2 0x00007ffff6a27f07 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6b31864 "*** %s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007ffff6ac2077 in __GI___fortify_fail (msg=msg@entry=0x7ffff6b3180a "buffer overflow detected") at fortify_fail.c:31
#4 0x00007ffff6ac0230 in __GI___chk_fail () at chk_fail.c:28
#5 0x00007ffff6ac1fe7 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
#6 0x00007ffff2a25dd4 in Add (pNsdsel=0x7fffdc002b50, pNsd=0x7fffded37860, waitOp=<optimized out>) at nsdsel_ptcp.c:91
#7 0x00007ffff5b8825c in RunSelect (sizeWorkset=128, workset=0x7ffff167c180, pThis=0x55555580c250) at tcpsrv.c:794
#8 Run (pThis=0x55555580c250) at tcpsrv.c:908
#9 0x00005555555b6184 in thrdStarter (arg=0x55555584c400) at ../threads.c:226
#10 0x00007ffff79b1dc5 in start_thread (arg=0x7ffff167d700) at pthread_create.c:308
#11 0x00007ffff6aaa76d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
This looks identical to GitHub issue https://github.com/rsyslog/rsyslog/issues/1728
Version-Release number of selected component (if applicable):
rsyslog-8.24.0-52.el7.x86_64
How reproducible:
Didn't try, always for customer.
Reproducer:
This is better done on two systems, but should work on a single system just fine.
1. Set up rsyslog to listen to remote TLS TCP syslog (https://access.redhat.com/articles/3549872)
2. Adjust the imtcp module settings to allow 8000 MaxSessions:
--8<--------8<--------8<--------8<--------8<--------8<--------8<--------8<------
module( load="imtcp"
# Update if handling large numbers of clients
MaxSessions = "8000"
StreamDriver.Name = "gtls"
StreamDriver.Mode = "1"
StreamDriver.AuthMode = "x509/name"
PermittedPeer = [ "syslog-client.example.com" ]
)
--8<--------8<--------8<--------8<--------8<--------8<--------8<--------8<------
3. Create a rsyslog.service override file with high LimitNOFILE, e.g. LimitNOFILE=16384
4. Start rsyslog server.
5. Run >1024 parallel TLS connections to the rsyslog server:
# let i=30000; while [ $i -le 31024 ]; do (nc -k -l -p $i | nc --ssl --ssl-cert /etc/pki/tls/syslog/cert.pem --ssl-key /etc/pki/tls/syslog/privkey.pem syslog-server.example.com 6514 &>/dev/null &); let i++; done
Result:
type=ANOM_ABEND msg=audit(1595610641.785:3328): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 pid=30285 comm="in:imtcp" reason="memory violation" sig=6
Jul 24 19:10:41 syslog-server.example.com systemd[1]: rsyslog.service: main process exited, code=killed, status=6/ABRT
Jul 24 19:10:41 syslog-server.example.com systemd[1]: Unit rsyslog.service entered failed state.
Jul 24 19:10:41 syslog-server.example.com systemd[1]: rsyslog.service failed.
Jul 24 19:10:42 syslog-server.example.com systemd[1]: rsyslog.service holdoff time over, scheduling restart.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (rsyslog bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2020:4073