Bug 1007379

Summary: rsyslog crashes
Product: Red Hat Enterprise Linux 6 Reporter: jhlavaty <JIRKA.HLAVATY>
Component: rsyslogAssignee: Radovan Sroka <rsroka>
Status: CLOSED WONTFIX QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3CC: dkopecek, jvymazal
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-17 12:44:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description jhlavaty 2013-09-12 11:30:32 UTC
Description of problem: rsyslog sometimes crashes with coredump, rsyslogd dead but pid file exists


gdb:

Version-Release number of selected component (if applicable): rsyslog-5.8.10-6.el6


warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
Core was generated by `/sbin/rsyslogd -i /var/run/syslogd.pid -c 5'.
Program terminated with signal 11, Segmentation fault.
#0  qDeqLinkedList (pThis=0x7f10dc263330, ppUsr=0x7f10d4fdbce8) at queue.c:566
566                *ppUsr = pEntry->pUsr;
(gdb) bt
#0  qDeqLinkedList (pThis=0x7f10dc263330, ppUsr=0x7f10d4fdbce8) at queue.c:566
#1  0x00007f10daec1105 in qqueueDeq (pThis=0x7f10dc263330, pWti=<value optimized out>) at queue.c:956
#2  DequeueConsumableElements (pThis=0x7f10dc263330, pWti=<value optimized out>) at queue.c:1461
#3  DequeueConsumable (pThis=0x7f10dc263330, pWti=<value optimized out>) at queue.c:1505
#4  0x0000000000000001 in ?? ()
#5  0x00007f10dc2635a8 in ?? ()
#6  0x00007f10d4fdbce0 in ?? ()
#7  0x0000001000dd0480 in ?? ()
#8  0x00007f10dc263440 in ?? ()
#9  0x00000000000003c0 in ?? ()
#10 0x00007f10dc263d98 in ?? ()
#11 0x00007f10dc2635a0 in ?? ()
#12 0x00007f10dc263d70 in ?? ()
#13 0x00000010dc264300 in ?? ()
#14 0x00007f10d4fdbcd8 in ?? ()
#15 0x00007f10dc263350 in ?? ()
#16 0x0000000052239cfd in ?? ()
#17 0x00007f10d0034560 in ?? ()
#18 0x0000000052239cfd in ?? ()
#19 0x00007f10d0009ac0 in ?? ()
#20 0x0000000000000000 in ?? ()





rsyslog.conf:

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$SystemLogRateLimitInterval 0

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;auth.!info;user.none;local0.none;local1.none;local2.none;local3.none;local4.none;local5.none;local6.none;local7.none     /var/log/messages

# The authpriv file has restricted access.
authpriv.*;auth.info                                    /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog

# Log cron stuff
cron.*                                                  /var/log/cron

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


### start forwarding rule ###
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName bb.aa.cc.cz # unique name prefix for spool files
$ActionQueueMaxDiskSpace 128m # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
$ActionQueueTimeoutEnqueue 1
*.* @@bb.aa.cc.cz:2526
### end forwarding rule ###

### start forwarding rule ###
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName xx.aa.cc.cz # unique name prefix for spool files
$ActionQueueMaxDiskSpace 128m # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
$ActionQueueTimeoutEnqueue 1
*.* @@xx.aa.cc.cz:2526
### end forwarding rule ###

Comment 2 Tomas Heinrich 2013-09-12 12:28:26 UTC
Hello Jiri,

how often does it crash? Once an hour / day / week ..?

Are you able to reproduce it reliably? (Even if it takes a couple of hours.)

Probably unrelated: the $WorkDirectory directive should be specified only once.

Comment 3 jhlavaty 2013-09-13 09:04:18 UTC
Hi Tomas,

This problem has more Linux server on our local network. Crashes was about 10 during the last six months. Reproduction accident is not possible. Looking for some dependences why falls occur (problem with log servers, network problems), but found none ..

thx for advice with  $WorkDirectory directive

Comment 4 jhlavaty 2013-09-13 13:37:12 UTC
Tomas, 

redhat has a special version of rsyslog? Official documentation of rsyslog recommend directive $WorkDirectory use multiple times. What is correct ?

Jirka

Comment 5 Tomas Heinrich 2013-09-13 14:27:22 UTC
(In reply to jhlavaty from comment #4)
> redhat has a special version of rsyslog?

No, apart from several patches.

> Official documentation of rsyslog
> recommend directive $WorkDirectory use multiple times. What is correct ?

You're right that it can be used multiple times, but since you're using the same directory with both, it is unnecessary.

Comment 6 RHEL Program Management 2013-10-14 02:13:19 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 8 Jiří Vymazal 2017-03-17 12:44:48 UTC
According to customer case, customer upgraded to rsyslog v7, which fixed the problem for them, closing