Escalated to Bugzilla from IssueTracker
Created attachment 351700 [details] rhel5-158.el5 - inotify: fix race
Created attachment 351701 [details] rhel5-158.el5 - inotify: remove debug code
*** Bug 520556 has been marked as a duplicate of this bug. ***
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
*** Bug 526853 has been marked as a duplicate of this bug. ***
We need to confirm that there is commitment to test for the resolution of this request during the RHEL 5.5 Test Phase, if it is accepted into the release. Please post a confirmation before Oct 23rd, 2009, including the contact information for testing engineers.
Description of problem: Customer has reported that the following messages in his production box. Mar 31 17:51:13 kernel: BUG: warning at fs/inotify.c:181/set_dentry_child_flags() (Tainted: PF ) Call Trace: [<ffffffff800ed831>] set_dentry_child_flags+0xef/0x14d [<ffffffff800ed8c7>] remove_watch_no_event+0x38/0x47 [<ffffffff800ed8ee>] inotify_remove_watch_locked+0x18/0x3b [<ffffffff800ed9dc>] inotify_rm_wd+0x7e/0xa1 [<ffffffff800edece>] sys_inotify_rm_watch+0x46/0x63 [<ffffffff8005d116>] system_call+0x7e/0x83 BUG: warning at fs/inotify.c:181/set_dentry_child_flags() (Tainted: PF ) Call Trace: [<ffffffff800ed831>] set_dentry_child_flags+0xef/0x14d [<ffffffff800ed8c7>] remove_watch_no_event+0x38/0x47 [<ffffffff800ed8ee>] inotify_remove_watch_locked+0x18/0x3b [<ffffffff800ed9dc>] inotify_rm_wd+0x7e/0xa1 [<ffffffff800edece>] sys_inotify_rm_watch+0x46/0x63 [<ffffffff8005d116>] system_call+0x7e/0x83 BUG: warning at fs/inotify.c:181/set_dentry_child_flags() (Tainted: PF ) Call Trace: [<ffffffff800ed831>] set_dentry_child_flags+0xef/0x14d [<ffffffff800ed8c7>] remove_watch_no_event+0x38/0x47 [<ffffffff800ed8ee>] inotify_remove_watch_locked+0x18/0x3b [<ffffffff800ed9dc>] inotify_rm_wd+0x7e/0xa1 [<ffffffff800edece>] sys_inotify_rm_watch+0x46/0x63 [<ffffffff8005d116>] system_call+0x7e/0x83 current Kernel: 2.6.18-92.1.13.el5 cat proc/cpuinfo | grep processor | wc -l 16 Soon after these messages, syslog shows: Mar 31 17:57:05 kernel: BUG: soft lockup - CPU#9 stuck for 10s! [java:9250] Mar 31 17:57:09 kernel: BUG: soft lockup - CPU#8 stuck for 10s! [java:9245] Mar 31 17:57:10 kernel: BUG: soft lockup - CPU#3 stuck for 10s! [snmpd:6501] Mar 31 17:57:11 kernel: BUG: soft lockup - CPU#7 stuck for 10s! [perl:15683] Mar 31 17:57:12 kernel: BUG: soft lockup - CPU#1 stuck for 10s! [vsftpd:15764] From SAR of March31, Load average on the system is on the higher side. This could be reason for unresponsiveness Following upstream patch seems to address the issue http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.24-rc6/2.6.24-rc6-mm1/broken-out/inotify-fix-race.patch Upstream patch from kernel bug 8938 seems to fit in cleanly with RHEL 5. http://bugzilla.kernel.org/show_bug.cgi?id=8938 How reproducible: not known Steps to Reproduce: Not known Actual results: inotify race is seen Expected results: inotify race needs to be fixed
in kernel-2.6.18-171.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified.
Installed kernel-2.6.18-171.el5 since 29 Oct, no sign of the message so far (it was quite frequent before)..
Same here, we've not seen any of the soft lockups or inotify warnings since we've installed the test kernel on Nov 3 on the machine where we've had soft lockups many times per day.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0178.html
*** Bug 545125 has been marked as a duplicate of this bug. ***