Bug 519192
| Summary: | rsyslog server cannot handle more than 1000 open files | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Dag Wieers <dag> |
| Component: | rsyslog | Assignee: | Tomas Heinrich <theinric> |
| Status: | CLOSED ERRATA | QA Contact: | BaseOS QE <qe-baseos-auto> |
| Severity: | urgent | Docs Contact: | |
| Priority: | medium | ||
| Version: | 5.3 | CC: | dag, fievelhat, pvrabec, sgrubb, tao, theinric |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2010-03-30 08:17:16 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Dag Wieers
2009-08-25 15:29:55 UTC
Configuration of our syslog server is quite simple: /etc/sysconfig/rsyslog ---- # Options to syslogd # -m 0 disables 'MARK' messages. # -rPortNumber Enables logging from remote machines. The listener will listen to the specified port. # -x disables DNS lookups on messages recieved with -r # See syslogd(8) for more details SYSLOGD_OPTIONS="-m 0 -r514 -t514,1000" # Options to klogd # -2 prints all kernel oops messages twice; once for klogd to decode, and # once for processing with 'ksymoops' # -x disables all klogd processing of oops messages entirely # See klogd(8) for more details KLOGD_OPTIONS="-x" ---- /etc/rsyslog.conf ---- # --- All remote files --- $template remote-messages,"/var/log/%HOSTNAME%/messages" $template remote-secure,"/var/log/%HOSTNAME%/secure" $template remote-maillog,"/var/log/%HOSTNAME%/maillog" $template remote-cron,"/var/log/%HOSTNAME%/cron" $template remote-spooler,"/var/log/%HOSTNAME%/spooler" $template remote-bootlog,"/var/log/%HOSTNAME%/boot.log" *.info;mail.none;authpriv.none;cron.none ?remote-messages authpriv.* ?remote-secure mail.* -?remote-maillog cron.* ?remote-cron uucp,news.crit ?remote-spooler local7.* ?remote-bootlog ---- We opened a problem ticket with our TAM for this as Issue 334499. Just for the public record. I wanted to add the following error messages we got from rsyslogd on our central rsyslog server today. We didn't see this error before, but now we have ulimit set to allow 5000 open files so maybe we hit the ulimit open files limit before and now we are hitting the select() limit instead. The system slowed down completely, trying to log on provided me with a login-prompt, but I got disconnected before a login via SSH happened. Colleagues tried to log in on the console, but failed as well. A forced reboot brought the system back, this was in the logfile on disk: ---- Sep 3 14:15:10 lg2log01 rsyslogd:select: Bad file descriptor Sep 3 14:15:59 lg2log01 rsyslogd:last message repeated 153422 times Sep 3 14:17:00 lg2log01 rsyslogd:last message repeated 294806 times Sep 3 14:18:01 lg2log01 rsyslogd:last message repeated 309974 times Sep 3 14:19:02 lg2log01 rsyslogd:last message repeated 297799 times Sep 3 14:20:03 lg2log01 rsyslogd:last message repeated 266527 times Sep 3 14:21:04 lg2log01 rsyslogd:last message repeated 273352 times Sep 3 14:22:05 lg2log01 rsyslogd:last message repeated 263318 times Sep 3 14:23:06 lg2log01 rsyslogd:last message repeated 241717 times ... Sep 3 17:22:02 lg2log01 rsyslogd:last message repeated 274251 times Sep 3 17:23:03 lg2log01 rsyslogd:last message repeated 284283 times Sep 3 17:39:06 lg2log01 rsyslogd: [origin software="rsyslogd" swVersion="2.0.6" x-pid="12291" x-info="http://www.rsyslog.com"][x-configInfo udpReception="Yes" udpPort="514" tcpReception="Yes" tcpPort="514"] restart Sep 3 17:39:06 lg2log01 rsyslog: rsyslogd startup succeeded ---- If this is happening with only 400 systems logging to one server, I wonder why there aren't more people reporting problems like this one, and the two others. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0213.html |