Bug 160861 - kernel oops on SCSI HBA hot-unplug
kernel oops on SCSI HBA hot-unplug
Status: CLOSED DUPLICATE of bug 160858
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.0
i686 Linux
medium Severity high
: ---
: ---
Assigned To: Kernel Maintainer List
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2005-06-17 16:53 EDT by Dan Duval
Modified: 2007-11-30 17:07 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-06-20 10:13:28 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan Duval 2005-06-17 16:53:54 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4

Description of problem:
While working on hardening the SCSI stack against surprise HBA
removal, we've come across an issue you might be interested in.

This issue is relevant any time a SCSI HBA is hot-unplugged while
I/O is going on to any of that HBA's devices.  Since more and more
platforms (including our own) are supporting PCI hotplug, we felt
that we should bring this problem to your attention as early as
possible.

In vanilla linux kernel 2.6.8.1 kernel, the routine scsi_remove_host()
(in drivers/scsi/hosts.c) begins as follows:

        scsi_host_cancel(shost, 0);
        scsi_proc_host_rm(shost);
        scsi_forget_host(shost);

In the patch for kernel 2.6.9, Mike Anderson changed this to the
following:

        scsi_forget_host(shost);
        scsi_host_cancel(shost, 0);
        scsi_proc_host_rm(shost);

The problem is that at least some of the functionality of
scsi_host_cancel() *must* come before at least some of the
functionality of scsi_forget_host().  The reason is the
following call chain:

    scsi_forget_host() ->

        /* Loop over host's associated devices... */
        scsi_remove_device() ->

            device_del() ->

                list_del_init(&dev->node);

This removes the SCSI device's sdev_gendev from the list rooted
at the HBA's host_gendev.children.  The result is that the HBA's
list of children ends up empty.

Later, we see the following call chain:

    scsi_host_cancel() ->

        device_for_each_child(..., scsi_device_cancel_cb)

The intent is to loop over the HBA's children and cancel
all outstanding I/O requests for each child (== SCSI device).
The list of children is, however, now empty, so no call to
scsi_device_cancel_cb() is ever made.

The practical effect of this is that the requests never get
canceled, so their timers are still active.  Thirty seconds
later, some request times out and gets queued to the error-
handling thread and that thread is awakened.  By this time,
however, we have torn down nearly all of the data structures
associated with the HBA.  In our testing, the error handler
ended up trying to requeue the request to the block layer,
whose elevator data structure had been freed.  Instant OOPS.

We reproduced this under RHEL AS4 by actually hot-unplugging
a QLogic qla1280 HBA while doing I/O to it.  A read-through of
the code suggests that, if hardware that allows hot-unplug is
not available, the problem can also be triggered by rmmod'ing
a LLD while I/O is going to any of its disks.

In kernel 2.6.10, the problem has been eliminated by modifying
scsi_host_cancel() so that it doesn't use the "children" list
to iterate over the devices, but rather uses the list rooted
at shost->__devices and linked via sdev->siblings.  This list is,
of course, not modified by device_del().

This problem may be related to:

    https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=133998

but I don't believe they are duplicates.

Version-Release number of selected component (if applicable):
kernel-2.6.9-1.648_EL

How reproducible:
Always

Steps to Reproduce:
1. Initiate some I/O to a SCSI device (via, e.g., "dd").

2. Hot-unplug the SCSI HBA or, failing that, rmmod the
   lower-level driver's kernel module.
 

Actual Results:  A kernel oops resulted.

Expected Results:  The system should degrade gracefully.  Outstanding I/O
request should be failed back to the application(s).

Additional info:
Comment 1 Dan Duval 2005-06-20 10:13:28 EDT

*** This bug has been marked as a duplicate of 160858 ***

Note You need to log in before you can comment on or make changes to this bug.