Bug 519203

Summary: Applications block when rsyslog has remote messages queued
Product: Red Hat Enterprise Linux 5 Reporter: Dag Wieers <dag>
Component: rsyslogAssignee: Tomas Heinrich <theinric>
Status: CLOSED ERRATA QA Contact: BaseOS QE <qe-baseos-auto>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 5.3CC: dag, jskrabal, mmatsuya, plyons, pvrabec, sgrubb, tao, theinric, zjw03124188
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-03-30 08:17:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 542631    

Description Dag Wieers 2009-08-25 16:00:41 UTC 
Description of problem:
We witnessed a severe problem on our central rsyslog server (too many open
files) as described in Bug 519192. This caused our local rsyslog daemon on all
our systems to stop accepting log messages from local applications (including sshd, mingetty) and as a consequence applications failed and nobody could log on to the server via ssh or console. Likely because messages were being discarded by our central syslog server or because tcp connections were not accepted (more likely).

This is a severe problem because a central syslog setup is a backup solution
for troubleshooting and security, but should not have any impact on servers
logging to it. And rsyslog should definitely have no impact on applications and logons.

On top of this, the local syslog daemon stopped logging local log messages to disk as described in Bug 519201. Complicating troubleshooting as a reboot would
render any recent logging missing from the logfile.

Version-Release number of selected component (if applicable):
rsyslog-2.0.6-1.el5

How reproducible:
See description.
Comment 1 Dag Wieers 2009-08-25 16:02:37 UTC 
The configuration on systems is pretty basic, and looks exactly like this:

/etc/rsyslog.conf
----
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

### Send all logging information to central syslog server
*.*                                                     @@syslog.domain.

### Send logging information to Envision
authpriv.*                                              @envision.domain.:514
----
Comment 2 Dag Wieers 2009-08-25 16:33:55 UTC 
We found the following public discussions confirming the bug from the author:

  http://lsts.adiscon.net/pipermail/rsyslog/2008-August/001006.html

  http://bugzilla.adiscon.com/show_bug.cgi?id=86
Comment 3 Dag Wieers 2009-08-25 16:38:50 UTC 
We opened a problem ticket with our TAM for this as Issue 334505
Comment 16 errata-xmlrpc 2010-03-30 08:17:05 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0213.html