Red Hat Bugzilla – Bug 848852
rsyslogd-2163 - epoll_ctl failed
Last modified: 2016-06-18 18:55:29 EDT
Description of problem:
our central loghost is routinely dumping out
Aug 15 09:45:11 log02 rsyslogd-2163: last message repeated 1662 times
Aug 15 09:45:11 log02 rsyslogd-2163:epoll_ctl failed on fd 136, id 0/0x7ff1d401fc10, op 1 with File exists
: File exists [try http://www.rsyslog.com/e/2163 ]
Version-Release number of selected component (if applicable):
that's a good question - we don't quite know how to reproduce it
It just happens pretty often recently. It only started following the rhel 6.3 upgrade.
it emits this message - specifically the last line which is not in syslog format and causes our log parser/analyzer to complain
it should not emit the message or rather should not have the problem causing the msg.
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Hi Seth, have you got some new information about this issue? Do we have something which could lead us to reproducer?
Sorry for the lag in reply - I must have missed the needinfo setting back in april.
I don't have any additional info. It continues to happen to us.
If it's still a problem, I'll see what I can do, but no promises. Would you be able to test a potential fix?
Yes - we can test out fixes. It sometimes takes a few days for the error to crop back up - but we can test out fixes.
We are seeing similar messages appear when there is an unusually high load on a rsyslog server that aggregates from several nodes. Its a RHEL6.4 system with kernel version is 2.6.32-279.19.1.el6.x86_64 and rsyslog-5.8.10-6.el6.x86_64.
rsyslogd-2163: epoll_ctl failed on fd 984, id 0/0x7f26e8377c40, op 1 with File exists: File exists [try http://www.rsyslog.com/e/2163 ]
Have any fixes for this bug proved successful?
I've also experienced this same issue with ryslog-5.8.10-7.el6_4.x86_64.
This is fixed in the version rsyslog-5.8.10-8.el6_4.x86_64. Please upgrade to it or onwards.
Also increase per user max open fd limit which is by default set to 1024 in RHEL.
root soft nofile 4096
root hard nofile 10240
As Susant has written, increasing the maximum number of file descriptors can be a viable workaround.
An updated package should be available in el6.5.
The upstream patch that should fix the cause for the "epoll_ctl" message is this:
But there's some other problem:
The side-effect seems to be that the daemon takes CPU utilization to 100% until a file descriptor frees up.
I need to investigate this further.
I will confirm your findings Tomas. I have this same core issue (epoll_ctl), and updating to rsyslog-5.8.10-8.el6_4.x86_64 does seem to "fix" the original problem, but it does cause the CPU usage to spike.
Because our instance is an HPC environment, we already had max open set higher than default in /etc/security/limits.conf
* soft nofile 32000
* hard nofile 100000
The original issue, as reported, has been already fixed in rsyslog-5.8.10-8 as noted in an earlier comment. This is the relevant patch:
Thus, I'm closing this bz.
There is a related issue, uncovered by the abovementioned patch, with spikes of CPU usage. I've filed it separately as bug #1077238.
Are there plans to incorporate this patch into the rsyslog5 package included with EL5 (currently at 5.8.12)?