Bug 505491

Initial bisect shows that this problem started occurring in the -141 kernel; prior to that, it booted just fine.  Further refinement shows that this is a hypervisor problem.  That is, a -141 dom0 kernel with a -141 hypervisor shows the problem, but a -141 dom0 kernel with a -140 hypervisor does not.  In the -141 release, the VT-d and AMD IOMMU code was added to the hypervisor, so something in there is likely causing the problem.

I'm continuing to investigate.

Chris Lalancette

Right.  So the trouble begins here:

    old_vector = domain_irq_to_vector(d, pirq);
    old_pirq = domain_vector_to_irq(d, vector);

    if ( (old_vector && (old_vector != vector) ) ||
         (old_pirq && (old_pirq != pirq)) )
    {
        dprintk(XENLOG_G_ERR, "dom%d: pirq %d or vector %d already mapped\n",
                d->domain_id, pirq, vector);
        return -EINVAL;
    }

in arch/x86/irq.c.  What's happening is that "old_vector" is being set to 226, meaning that we fall into the if() statement, and then all is lost.  Now, domain_irq_to_vector is actually a macro in include/asm-x86/irq.h:

#define domain_irq_to_vector(d, irq) ((d)->arch.pirq_vector[irq] ?: \
                                      IO_APIC_IRQ(irq) ? 0 : LEGACY_VECTOR(irq))

In point of fact, that macro *was* changed with the VT-d stuff; it used to be a much simpler:

#define domain_irq_to_vector(d, irq) ((d)->arch.pirq_vector[(irq)])

Now let's look at what is going on.  During bootup, if we detect an IOAPIC in the system (which we do), then we run through arch/x86/io_apic.c:setup_IO_APIC().  In there, we end up setting "io_apic_irqs" to 0xffffffff, and calling "init_IO_APIC_traps()".  init_IO_APIC_traps looks like this:

static inline void init_IO_APIC_traps(void)
{
    int irq;
    /* Xen: This is way simpler than the Linux implementation. */
    for (irq = 0; irq < 16 ; irq++)
        if (IO_APIC_IRQ(irq) && !IO_APIC_VECTOR(irq))
            make_8259A_irq(irq);
}

So, it runs through all of the legacy IRQs, and sees if they are managed by the IO_APIC.  Now, since we earlier said that all IRQs are managed by the IO_APIC, the first check is true for all legacy vectors.  However, the second check (!IO_APIC_VECTOR()) is only true for irq #2 (why?  I need to figure this out).  That means that we end up masking irq #2 out of the the io_apic_irqs.  Later on, when we go to map that IRQ into domain 0, it fails because that IRQ is masked out, and so the domain_irq_to_vector() ends up returning 0xe2, or a LEGACY_VECTOR, and then the "if (old_vector)" test fires.

Chris Lalancette

I sent this mail to virt-intel-list, and while it duplicates some of Comment #3, I think it explains the situation a bit better:

Hello all,
     Internal QE just pointed us at a regression on old hardware, which I
tracked down to the VT-d patches.  The hardware in questions is an old AMD
Athlon machine, pre-64bit.  Prior to the VT-d patches going in (in the
2.6.18-141xen kernel), this machine would boot and run Xen just fine.  However,
after the VT-d patches, this machine no longer boots.  The problem manifests
itself as a hang during dom0 boot, and you get the following messages on the
console:

Freeing SMP alternatives: 13k freed
ACPI: Core revision 20060707
ENABLING IO-APIC IRQs
(XEN) ioapic_guest_write: apic=0, pin=0, old_irq=-1, new_irq=0
(XEN) ioapic_guest_write: old_entry=00010000, new_entry=000009f0
(XEN) ioapic_guest_write: Attempt to add IO-APIC pin for in-use IRQ!
(XEN) irq.c:794: dom0: pirq 2 or vector 144 already mapped
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=-1, new_irq=4
(XEN) ioapic_guest_write: old_entry=00010000, new_entry=000009e4
(XEN) ioapic_guest_write: Attempt to add IO-APIC pin for in-use IRQ!

The important (different) bit here is that "pirq 2 or vector 144 already mapped"
message.  I tracked it down to this change:

-#define domain_irq_to_vector(d, irq) ((d)->arch.pirq_vector[(irq)])
+#define domain_irq_to_vector(d, irq) ((d)->arch.pirq_vector[irq] ?: \
                                      IO_APIC_IRQ(irq) ? 0 : LEGACY_VECTOR(irq))

in the hypervisor in include/asm-x86/irq.h.  What seems to be happening is that
during ACPI parse, we find out that we have an IOAPIC, so we set acpi_ioapic to
1.  Then, during IOAPIC setup in setup_IO_APIC(), we set the io_apic_irqs mask
to ~0, meaning that we think all IRQs are routed through the IOAPIC (because
that's what ACPI told us).  However, during the rest of IOAPIC setup, in
particular during setup_IO_APIC_irqs(), (I think) we find that this is not the
case, and IRQ 2 isn't actually routed through the IOAPIC.  Because of this,
init_IO_APIC_traps() masks IRQ 2 out of io_apic_irqs.

Now, dom0 starts up and starts mapping pirq's.  When it gets to pirq 2,
map_domain_pirq() does:

old_vector = domain_irq_to_vector(d, pirq);

Looking at the "new" version of the domain_irq_to_vector macro above, though, we
see that d->arch.pirq_vector[2] == 0 (since we've not tried to map this yet).
Therefore, we check the next part, which is IO_APIC_IRQ(2), but from the above,
we know that pirq 2 is *not* mapped through the IOAPIC.  Because of that, we
drop through and return LEGACY_VECTOR(2), which ends up being 0xe2.  This return
value causes this check:

    if ( (old_vector && (old_vector != vector) ) ||
         (old_pirq && (old_pirq != pirq)) )
    {
        dprintk(XENLOG_G_ERR, "dom%d: pirq %d or vector %d already mapped\n",
                d->domain_id, pirq, vector);
        return -EINVAL;
    }

to fire, and hence the mapping (and the boot) to fail.

I'm not entirely sure what the fix would be.  It seems to me that the
domain_irq_to_vector macro needs to take into account pirqs which aren't routed
through the IOAPIC, but also aren't "LEGACY" vectors.  Does anybody have
thoughts about how to go about fixing this?  For reference, this is also broken
with an upstream hypervisor.

Chris Lalancette

Created attachment 348102 [details]
xm dmesg from -128 xen kernel

Created attachment 348103 [details]
dmesg from -128 dom0

Created attachment 348104 [details]
xm dmesg from -153 xen kernel (collected via serial console)

Created attachment 348105 [details]
dmesg from -153 dom0

Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
On older 32-bit systems with only 1 CPU and an IOAPIC, the 32-bit Xen dom0 kernel may fail to boot with message similar to:

(XEN) irq.c:794: dom0: pirq 2 or vector 144 already mapped
(XEN) io_apic.c:2207: 
(XEN) ioapic_guest_write: apic=0, pin=2, old_irq=-1, new_irq=4
(XEN) ioapic_guest_write: old_entry=00010000, new_entry=000009e4
(XEN) ioapic_guest_write: Attempt to add IO-APIC pin for in-use IRQ!

To workaround this problem, pass the option "noapic" to the hypervisor while booting the kernel.  This bug exists in the RHEL-5.4 beta kernel, but should be addressed before RHEL-5.4 Final.

in kernel-2.6.18-156.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Please do NOT transition this bugzilla state to VERIFIED until our QE team
has sent specific instructions indicating when to do so.  However feel free
to provide a comment indicating that this fix has been verified.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-1243.html

Clearing a needinfo request.

Chris Lalancette