Description of problem: see below


Version-Release number of selected component: kernel 2.6.18-164.el5


How reproducible: dependent on system configurations.  If a system
is configured such that the problem occurs, it will occur every time
on that system.


Steps to Reproduce: see description.  Involves hot-plugging and
-unplugging MSI/MSI-X -capable PCI hardware.


Actual results: kernel appears to hang solidly.  Eventually, on
systems so equipped, an external watchdog will NMI the system.


Expected results: hot-pluggable PCI hardware should be added to
and removed from a system without triggering a crash.


Additional info:

	Affected architecture: x86_64 (and possibly i386 -- not tested)


Description:

0. On x86_64 systems that use the PIT clock to generate ticks,
the timer interrupt is always allocated vector (IDT index) 0x31
(== FIRST_DEVICE_VECTOR).  This slot in the IDT is then set to
point to the IRQ0 interrupt stub IRQ0x00_interrupt.

This is done relatively early in the boot process.

1. The central vector allocator for the kernel is

        arch/x86_64/kernel/io_apic.c:assign_irq_vector()

This routine is given a range of vectors from FIRST_DEVICE_VECTOR
to FIRST_SYSTEM_VECTOR-1 that it can hand out.  It does so starting
with 0x39, and marching in strides of 8 up the range: 0x41, 0x49, ...
When it reaches the end, it wraps around to the beginning of the
range and begins allocating the vectors that are equal to 2 (mod
8), and so forth.  In this way, it will eventually cover all the
vectors in its range.

2. Allocation of interrupts for MSI and MSI-X PCI devices is done by

        drivers/pci/msi.c:assign_msi_vector()

Each time it needs a new interrupt vector, this routine calls
assign_irq_vector() to get one.  This occurs until it obtains
vector LAST_DEVICE_VECTOR (which is FIRST_SYSTEM_VECTOR-1, i.e.,
the end of the range), at which time it switches to a different mode
where it no longer allocates new vectors, but instead attempts to
re-allocate vectors that have previously been freed.  The routine
remains in this mode for the rest of the system lifetime.

3. The problem arises because assign_msi_vector() is not the only
consumer of vectors from assign_irq_vector().  In particular,
the ACPI code does its own independent allocation through

        arch/x86_64/kernel/io_apic.c:gsi_irq_sharing()

which also calls assign_irq_vector().

So, if things happen in just the wrong order, gsi_irq_sharing() can
consume vector LAST_DEVICE_VECTOR.  This prevents assign_msi_vector()
from ever seeing LAST_DEVICE_VECTOR, which means that it never
switches modes.  Instead, it continues to call assign_irq_vector()
for every allocation.

4. Because the timer vector 0x31 is within the range that
assign_irq_vector() can allocate, eventually it will do so.
This results in changing the corresponding IDT entry to point at a
different entry point (IRQ) among the interrupt stubs.  At this point,
the timer interrupt handler is "orphaned", and the system stops
processing timer ticks.  This tends to have an adverse effect on
system performance.

5. This problem is particular to hot-pluggable PCI devices that
use MSI or MSI-X. In some cases, it can take only one hot-unplug and
-plug to trigger the condition.

Since occurrence depends on the exact configuration of system hardware,
and on the order in which it is scanned, the problem either happens
every time or doesn't happen at all.  It's conceivable that simply
swapping two PCI cards between slots could turn a "good" configuration
into a "bad" one.

6. The most recent upstream kernels, including the ones being
used for RHEL6, have introduced two mechanisms for preventing
this problem.

First, the range of vectors available for devices has been
restricted so that, now, the timer vector falls outside the range.
assign_irq_vector() therefore will never "commandeer" the timer
vector.

Note that this change alone doesn't address the underlying problem.

Second, a 256-bit bitmap "used_vectors" has been introduced.
assign_irq_vector() checks this bitmap when asked to allocate a new
vector, and will not reallocate a vector that is already marked as
"used".

If asked to guess, I would say that it would be reasonably
straightforward to backport the appropriate upstream changes to
this kernel.