Description of problem: Panic is occurring in the I/O completion interrupt handling for the character interface driver (sg). (race condition) The sg i/o completion bottom half handler code in sg_cmd_done_bh() is waking up a process via wake_up_interruptible() before calling kill_fasync() to cleanup any async helper structure. Unfortunately, under rare conditions on a multi-processor host, the process awoken via wake_up_interruptible() may be able to both call sg_read() to retrieve the completed sg i/o and call sg_release() to terminate the established sg session before the sg i/o completion handling code gets to call kill_fasync().b In this case, the sg_release() code will de-allocate the memory used for the sg session data structure via sg_fasync() calling fasync_helper(), possibly even returning the physical page and unmapping the virtual address to the page. Any attempt to de-reference through this address in kill_fasync() will panic the host. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 101559 [details] patch for RHEL 3.0 U2 sg.cb
The patch looks okay to me. I would like to have it reviewed on the linux-scsi list, and incorporated upstream if possible. Would the author of this patch be willing to post this to linux-scsi? (Use a unified diff when posting to the Linux lists.) If not, I will post it. If it passes review, the patch will be in U4. Tom
Would you please take the lead on this issue and post the patch for me Tom? Thanks
Steven Tweedie reviewed the patch, and does not think it is a sufficient fix: It's an interruptible task, so there's nothing at all to stop user-space from signalling or timing out or otherwise continuing on its own independently of the wake_up_interruptible(). In that case, there's nothing to stop the race happening *before* we take the copy of sfp->async_qp. All we're doing here is fixing the most likely cause of the wakeup/read()/sg_release(). The patch seems to be saying that there are plenty of other ways of reaching the same race which are not addressed by the patch. Don't we really need to move the kill_fasync() up to within the locking, before we let go of the command completely? --- I am looking at whether we can find a suitable solution before U4 freezes. Tom
A patch has been proposed upstream, and looks like it will be accepted. http://marc.theaimsgroup.com/?l=linux-scsi&m=109936088901128&w=2 Please test to confirm this fixes your problem. This is a candidaate for RHEL 3 U5.
This bug was found when PowerPath contained a volume manage and the problem isn't being replicated. The PowerPath team is reviewing the code and will provide an update.
To clarify, there is no longer a volume manager included in the PowerPath package and the problem isn't being replicated.
A fix for this problem has just been committed to the RHEL3 U5 patch pool this evening (in kernel version 2.4.21-27.13.EL).
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2005-294.html