Bug 519203 - Applications block when rsyslog has remote messages queued
Summary: Applications block when rsyslog has remote messages queued
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: rsyslog
Version: 5.3
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Tomas Heinrich
QA Contact: BaseOS QE
URL:
Whiteboard:
Depends On:
Blocks: 542631
TreeView+ depends on / blocked
 
Reported: 2009-08-25 16:00 UTC by Dag Wieers
Modified: 2018-10-27 15:53 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 08:17:05 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0213 normal SHIPPED_LIVE rsyslog bug fix update 2010-03-29 12:31:05 UTC

Description Dag Wieers 2009-08-25 16:00:41 UTC
Description of problem:
We witnessed a severe problem on our central rsyslog server (too many open
files) as described in Bug 519192. This caused our local rsyslog daemon on all
our systems to stop accepting log messages from local applications (including sshd, mingetty) and as a consequence applications failed and nobody could log on to the server via ssh or console. Likely because messages were being discarded by our central syslog server or because tcp connections were not accepted (more likely).

This is a severe problem because a central syslog setup is a backup solution
for troubleshooting and security, but should not have any impact on servers
logging to it. And rsyslog should definitely have no impact on applications and logons.

On top of this, the local syslog daemon stopped logging local log messages to disk as described in Bug 519201. Complicating troubleshooting as a reboot would
render any recent logging missing from the logfile.

Version-Release number of selected component (if applicable):
rsyslog-2.0.6-1.el5

How reproducible:
See description.

Comment 1 Dag Wieers 2009-08-25 16:02:37 UTC
The configuration on systems is pretty basic, and looks exactly like this:

/etc/rsyslog.conf
----
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

### Send all logging information to central syslog server
*.*                                                     @@syslog.domain.

### Send logging information to Envision
authpriv.*                                              @envision.domain.:514
----

Comment 2 Dag Wieers 2009-08-25 16:33:55 UTC
We found the following public discussions confirming the bug from the author:

  http://lsts.adiscon.net/pipermail/rsyslog/2008-August/001006.html

  http://bugzilla.adiscon.com/show_bug.cgi?id=86

Comment 3 Dag Wieers 2009-08-25 16:38:50 UTC
We opened a problem ticket with our TAM for this as Issue 334505

Comment 16 errata-xmlrpc 2010-03-30 08:17:05 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0213.html


Note You need to log in before you can comment on or make changes to this bug.