Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
With an rsyslog server configured with the following characteristics is stressed, with connectivity broken to the remote rsyslog server, it experiences a segfault condition
- imfile plugin with 3 separate input files defined
- Disk Assisted configuration
$WorkDirectory /var/lib/rsyslog
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName emergency_syslog
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
$ActionQueueSize 1000
$ActionQueueLowWaterMark 1 # the default
$ActionQueueHighWaterMark 100 # the default
$ActionQueueDiscardMark 9750 # the default
$ActionQueueTimeoutEnqueue 0
$ActionQueueMaxFileSize 10M
$ActionQueueMaxDiskSpace 20M
- Log Forwarding via TCP remote machine
The failure is due to NULL pointer dereference as visible below:
Thread 1 (Thread 0x7f41bd061700 (LWP 19807)):
#0 qDelLinkedList (pThis=0x7f41c4bf68d0) at queue.c:583
#1 0x00007f41c2d216ec in DoDeleteBatchFromQStore (pThis=0x7f41c4bf68d0, nElem=16) at queue.c:1340
#2 0x00007f41c2d2419d in DeleteBatchFromQStore (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1368
#3 DeleteProcessedBatch (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1428
#4 DequeueConsumableElements (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1457
#5 DequeueConsumable (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1505
#6 0x00007f41c2d2425d in DequeueForConsumer (pThis=<value optimized out>, pWti=0x7f41c4bf7350) at queue.c:1642
#7 ConsumerDA (pThis=<value optimized out>, pWti=0x7f41c4bf7350) at queue.c:1751
#8 0x00007f41c2d1ceb6 in wtiWorker (pThis=0x7f41c4bf7350) at wti.c:313
#9 0x00007f41c2d1c9aa in wtpWorker (arg=0x7f41c4bf7350) at wtp.c:387
#10 0x00007f41c26a0aa1 in start_thread (arg=0x7f41bd061700) at pthread_create.c:301
#11 0x00007f41c1dcbaad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
The specific instruction underway at the time is:
(gdb) disassemble /m qDeqLinkedList
Dump of assembler code for function qDeqLinkedList:
560 {
561 qLinkedList_t *pEntry;
562 DEFiRet;
563
564 pEntry = pThis->tVars.linklist.pDeqRoot;
0x00007fb575dfa080 <+0>: mov 0x240(%rdi),%rax
565 ISOBJ_TYPE_assert(pEntry->pUsr, msg);
566 *ppUsr = pEntry->pUsr;
=> 0x00007fb575dfa087 <+7>: mov 0x8(%rax),%rdx
0x00007fb575dfa08e <+14>: mov %rdx,(%rsi)
<snip>
Where RAX is 0:
(gdb) p $rax
$2 = 0
Version-Release number of selected component (if applicable):
rsyslog-5.8.10-10.el6_6.x86_64
How reproducible:
Easily, in the end customer environment
Steps to Reproduce:
1. Configure the server in the above described manner
2. Start a stress-test which emits messages via logger
3. Disconnect the backing remote machine so that it is no longer above to receive messages
Actual results:
Segfault
Expected results:
Continued operation
Additional info:
Hello Kyle,
we start providing rsyslog7 as a replacement for rsyslog5 which was very old code base not supported by upstream.
We made a decision with product management that we will stop updating the old package and start maintaining the new one.(rsyslog7) [1]
We recommend updating rsyslog to the new package.[2]
I'm setting NAK on this request based on that ^^^ information.
Peter.
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=869600#c48
[2]: https://access.redhat.com/solutions/282663
Comment 3Red Hat Bugzilla Rules Engine
2017-09-14 15:25:46 UTC
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.
I totally understand and support the idea of dropping old software but I don't think saying:
"All users who require rsyslog7 are advised to install these new packages. All
users of rsyslog are advised to migrate to these new packages."
is in any way equivalent to:
"we are no longer supporting the rsyslog package in el6"
which is what I believe you are saying here. the migration path that was punted down to customers is untenable for large installations. running yum-shell on hundreds or thousands of hosts manually can't possibly be what you're expecting out of customers, can it? if rsyslog 5.x and the resulting packages really are unsupported the upgrade to 7 should be handled by the default packages, not passed down to customers.
I doubt any change will be made here but I feel this is being really poorly handled and should be reconsidered/escalated back to management.
essentially what you have done is silently make a core stack component unsupported. even worse, you've chosen a very common logging implementation to do it.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:1863
Description of problem: With an rsyslog server configured with the following characteristics is stressed, with connectivity broken to the remote rsyslog server, it experiences a segfault condition - imfile plugin with 3 separate input files defined - Disk Assisted configuration $WorkDirectory /var/lib/rsyslog $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName emergency_syslog $ActionQueueSaveOnShutdown on $ActionResumeRetryCount -1 $ActionQueueSize 1000 $ActionQueueLowWaterMark 1 # the default $ActionQueueHighWaterMark 100 # the default $ActionQueueDiscardMark 9750 # the default $ActionQueueTimeoutEnqueue 0 $ActionQueueMaxFileSize 10M $ActionQueueMaxDiskSpace 20M - Log Forwarding via TCP remote machine The failure is due to NULL pointer dereference as visible below: Thread 1 (Thread 0x7f41bd061700 (LWP 19807)): #0 qDelLinkedList (pThis=0x7f41c4bf68d0) at queue.c:583 #1 0x00007f41c2d216ec in DoDeleteBatchFromQStore (pThis=0x7f41c4bf68d0, nElem=16) at queue.c:1340 #2 0x00007f41c2d2419d in DeleteBatchFromQStore (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1368 #3 DeleteProcessedBatch (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1428 #4 DequeueConsumableElements (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1457 #5 DequeueConsumable (pThis=0x7f41c4bf68d0, pWti=<value optimized out>) at queue.c:1505 #6 0x00007f41c2d2425d in DequeueForConsumer (pThis=<value optimized out>, pWti=0x7f41c4bf7350) at queue.c:1642 #7 ConsumerDA (pThis=<value optimized out>, pWti=0x7f41c4bf7350) at queue.c:1751 #8 0x00007f41c2d1ceb6 in wtiWorker (pThis=0x7f41c4bf7350) at wti.c:313 #9 0x00007f41c2d1c9aa in wtpWorker (arg=0x7f41c4bf7350) at wtp.c:387 #10 0x00007f41c26a0aa1 in start_thread (arg=0x7f41bd061700) at pthread_create.c:301 #11 0x00007f41c1dcbaad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 The specific instruction underway at the time is: (gdb) disassemble /m qDeqLinkedList Dump of assembler code for function qDeqLinkedList: 560 { 561 qLinkedList_t *pEntry; 562 DEFiRet; 563 564 pEntry = pThis->tVars.linklist.pDeqRoot; 0x00007fb575dfa080 <+0>: mov 0x240(%rdi),%rax 565 ISOBJ_TYPE_assert(pEntry->pUsr, msg); 566 *ppUsr = pEntry->pUsr; => 0x00007fb575dfa087 <+7>: mov 0x8(%rax),%rdx 0x00007fb575dfa08e <+14>: mov %rdx,(%rsi) <snip> Where RAX is 0: (gdb) p $rax $2 = 0 Version-Release number of selected component (if applicable): rsyslog-5.8.10-10.el6_6.x86_64 How reproducible: Easily, in the end customer environment Steps to Reproduce: 1. Configure the server in the above described manner 2. Start a stress-test which emits messages via logger 3. Disconnect the backing remote machine so that it is no longer above to receive messages Actual results: Segfault Expected results: Continued operation Additional info: