Bug 490477

Created attachment 335367 [details]
dmidecode output

Does it work if you disable legacy USB keyboard/mouse emulation in the BIOS?

(In reply to comment #2)
> Does it work if you disable legacy USB keyboard/mouse emulation in the BIOS?  

Yes the kernel boots this way but then the system hangs at starting udev.

A BIOS bug then. I suspect the hang at starting udev is going to be something different?

Created attachment 336751 [details]
smaller version of error message image

Fixes that went into Fedora 11 are now in 2.6.29.1-19 and later kernels.

(In reply to comment #4)
> A BIOS bug then. I suspect the hang at starting udev is going to be something
> different?  

Updated to BIOS 0302 but it did not fix it (well changelog says "new cpu support")

(In reply to comment #6)
> Fixes that went into Fedora 11 are now in 2.6.29.1-19 and later kernels. 

Will test when the build show up in koji ;)

(In reply to comment #7)
> (In reply to comment #4)
> > A BIOS bug then. I suspect the hang at starting udev is going to be something
> > different?  
> 
> Updated to BIOS 0302 but it did not fix it (well changelog says "new cpu
> support")
> 
> (In reply to comment #6)
> > Fixes that went into Fedora 11 are now in 2.6.29.1-19 and later kernels. 
> 
> Will test when the build show up in koji ;)  

Still same/similar results.
Sometimes it hangs when trying to set up the hpet. (booting with nohpet results into always hanging at the same place, see screenshot attached before)

The hang at boot is going to be a BIOS bug. The BIOS is using DMA for legacy keyboard/mouse emulation and doesn't tell us about it. So its DMA gets stopped (those are the faults you see in the photo). And then it hangs up in SMM mode eventually, because BIOSes are written by monkeys.

I'm more interested in the 'hang when starting udev' that you reported, after you turned off the legacy keyboard/mouse emulation. Did you ever get any more information about that?

How about we close this as NOTABUG/WONTFIX then and file the udev bug separately?

That makes sense.

*** Bug 499614 has been marked as a duplicate of this bug. ***

Created attachment 342840 [details]
potential workaround for broken bios

Hm, perhaps we could work around this with something like the attached patch

I tried to fix this problem with the potential fix attached to this bug, but it didn't work out. The system still hangs at "starting udev" for a while and then produces a lot of errors and then gets into a loop of endlessly reporting

"/etc/rc.d/rc.sysinit: line 824: /bin/usleep: Input/output error"

I'm going to attach my captured log to this bug.

Created attachment 343924 [details]
Logfile from the console

Crash produced with a patched kernel.

Rainer, the patch would do nothing for you -- it just disables the IOMMU for Need Real Name's machine based on the DMI identification.

If you have turned off the legacy keyboard/mouse emulation, you've worked around the BIOS bug which was preventing the machine from booting. Now you're left with an entirely more interesting problem, which may well be a Linux bug.

Please can you boot with 'mem=2G' and see if you can still reproduce the problem when you don't have any memory above physical address 0x100000000 ?

Hi David,

yes you're right. I just applied the patch without looking at it. :-) But if it just disables the IOMMU then its not helping since the goal is to get the IOMMU working. :-)

I tried out with mem=2G and had the same problem on that machine. But I still believe that its somewhat BIOS related. Maybe we should reopen

https://bugzilla.redhat.com/show_bug.cgi?id=499614 

since it looks like that this but is no longer been considered to be a duplicate of that bug. :-)

Regards
Rainer

I believe the two systems are behaving similarly. First there's a BIOS bug which makes it crash if you enable legacy keyboard/mouse emulation. That's the bug which was originally reported, and it's unfixable except by promoting an attitude of violence towards BIOS engineers. So strictly speaking the bugs should be closed.

We're actually now discussing a _separate_ bug, which is the 'long hang in udev' which both of you seem to be seeing, although one of you (Rainer) has now reported that if you wait long enough (how long?), it eventually finishes and you see other interesting stuff.

Need-Real, can you reproduce that observation? How long did you leave it at 'starting udev' before you gave up and reset it?

(In reply to comment #18)
> I believe the two systems are behaving similarly. First there's a BIOS bug
> which makes it crash if you enable legacy keyboard/mouse emulation. That's the
> bug which was originally reported, and it's unfixable except by promoting an
> attitude of violence towards BIOS engineers. So strictly speaking the bugs
> should be closed.
> 
> We're actually now discussing a _separate_ bug, which is the 'long hang in
> udev' which both of you seem to be seeing, although one of you (Rainer) has now
> reported that if you wait long enough (how long?), it eventually finishes and
> you see other interesting stuff.
> 
> Need-Real, can you reproduce that observation? How long did you leave it at
> 'starting udev' before you gave up and reset it?  

So tested this again it.
I waited ~5 min and nothing happened (seems to hang forever). While udev was trying to start the HDD led was constantly "on" and the (USB) keyboard was dead (could not even toggle the numlock).

http://www.scan.co.uk/Products/Asus-P6T-Deluxe-V2-Intel-X58-S1366-PCI-E-20-%28x16%29-Triple-DDR3-2000%28OC%29-1866%28OC%29-1600%28OC%29-ATX

Need-Real, you have one of these?

(In reply to comment #20)
> http://www.scan.co.uk/Products/Asus-P6T-Deluxe-V2-Intel-X58-S1366-PCI-E-20-%28x16%29-Triple-DDR3-2000%28OC%29-1866%28OC%29-1600%28OC%29-ATX
> 
> Need-Real, you have one of these?  

Yes that's the board that I am using and have issues with.

Try blacklisting the wireless devices.
I've seen problems w/my wireless device (that other wired-nic's don't);
applying various upstream (wireless driver) patches stopped the kernel
crashes I saw on my Dell laptop, but it would leave it hanging
when the wireless went to do dhcp (but phase that connects
to wireless router worked).

(In reply to comment #18)
> I believe the two systems are behaving similarly. First there's a BIOS bug
> which makes it crash if you enable legacy keyboard/mouse emulation. That's the
> bug which was originally reported, and it's unfixable except by promoting an
> attitude of violence towards BIOS engineers. So strictly speaking the bugs
> should be closed.

I doubt that you can solve a techical problem by adding a social layer to it. :-)

Since the kernel reports 

DMAR:Unknown DMAR structure type

in my case I'm going to discuss this with BIOS development. There must be a technical reason *why* this message is there and digging into this should help somehow. A colleauge also offered to run a VT-d test for Windows on that machine so that we can verify the integrity of the data provided by the BIOS. 

Probably also BIOS deveolpment should have some data verification tools from Intel. Unfortunately this bug is not the one with the highest priority, but I keep trying to get a root cause for the misbehaviour. 

Regards
Rainer

The unknown structure is probably ATSR and entirely harmless -- it's not going to be related to the problem. Running the VT-d test suite would be useful.

(In reply to comment #22)
> Try blacklisting the wireless devices.
> I've seen problems w/my wireless device (that other wired-nic's don't);
> applying various upstream (wireless driver) patches stopped the kernel
> crashes I saw on my Dell laptop, but it would leave it hanging
> when the wireless went to do dhcp (but phase that connects
> to wireless router worked).  

Can you file a separate bug for that please? And if it happens only with the IOMMU enabled, please CC me.

Forgot to add this information here:

While trying to debug the udev hang it turned out that the harddisk dma goes AWOL which results into a lot of input/output errors.

(In reply to comment #26)
> Forgot to add this information here:
> 
> While trying to debug the udev hang it turned out that the harddisk dma goes
> AWOL which results into a lot of input/output errors.  


That's kind of what I suspected -- but _when_ does it happen? It doesn't happen when we first enable the IOMMU; the disk is still working after that.

Can you try the 2.6.31-rc1 kernel with 'iommu=pt' ?

I have one of these now, and have reproduced the problem. It works fine with the IOMMU on (as long as Legacy USB support is disabled in the BIOS), and the hang "in udev" is caused by loading the snd-hda-intel driver. Blacklisting that makes it work fine. Will investigate further...

Pressing the reset button after the crash doesn't seem to work -- I have to power cycle the box. The BIOS claims to have a separate IOMMU just for the sound device, and I think it's telling the truth -- if I hack the code to let the sound fall under the catch-all device, then sound DMA fails with a 'Present bit in root entry is clear' fault as we would expect.

So the problem seems to happen when we set up and use a DMA mapping on the second IOMMU unit. I'll see if I can get help from some chipset folks.

Rainer, is your problem also avoided by blacklisting the snd-hda-intel driver?

If I boot with 'iommu=pt', then it works fine (I didn't try actual sound output, but it probes the codec OK and the system doesn't die).

If I hack the code to not use hardware pass-through, but instead use the software 1:1 mapping, it fails. I'm beginning to suspect a hardware problem...

(In reply to comment #31)
> If I boot with 'iommu=pt', then it works fine (I didn't try actual sound
> output, but it probes the codec OK and the system doesn't die).
> 
> If I hack the code to not use hardware pass-through, but instead use the
> software 1:1 mapping, it fails. I'm beginning to suspect a hardware problem...  

What kind of hardware problem exactly?

I don't know. But so far I've observed that: 

 - Hardware pass-through on the sound IOMMU works fine.
 - Software pass-through (where we set up a 1:1 mapping in the page tables for
   all of memory as reported by E820) fails as you described, which is that
   _ALL_ DMA in the system, even that on the other IOMMU, stops dead.

There's very little scope for a software bug to be causing this -- or even a BIOS bug AFAICT. Unless the BIOS is describing the hardware wrongly in some way such that it _almost_ works...

(In reply to comment #33)
> I don't know. But so far I've observed that: 
> 
>  - Hardware pass-through on the sound IOMMU works fine.
>  - Software pass-through (where we set up a 1:1 mapping in the page tables for
>    all of memory as reported by E820) fails as you described, which is that
>    _ALL_ DMA in the system, even that on the other IOMMU, stops dead.
> 
> There's very little scope for a software bug to be causing this -- or even a
> BIOS bug AFAICT. Unless the BIOS is describing the hardware wrongly in some way
> such that it _almost_ works...  

OK, is it possible for you (with your Intel Hat on) to get some contact to ASUS to get any information from them about this?

My contact attempts went straight to /dev/null

I think we can probably manage that. I'm talking to the chipset folks right now, and if we decide it's a board layout or BIOS issue then I think they have some way of making contact.

I tried hacking the code to treat sound devices like we do graphics devices when booted with 'intel_iommu=igfx_off' -- i.e. ignore the dedicated IOMMU unit completely, and do no mapping for the PCI device in question.

The board locks up at boot time, when enabling the _other_ IOMMU.

Created attachment 356205 [details]
Quirk to use passthrough mode for sound

This patch, which applies on top of commit 19943b0e at git://git.infradead.org/iommu-2.6.git, should work around the problem by using hardware passthrough mode for the sound device.

OK, I have built your tree with the patch applied on top of it and it seems to work fine (with usb legacy off).

System booted and I am currently writing this comment while running it with IOMMU on ;)

Now we need some kind of workaround for the USB issue (if it is possible to do anything about it in the kernel at all).

The USB issue is worked around by the patch at https://lists.linux-foundation.org/pipermail/iommu/2009-August/001657.html

(In reply to comment #39)
> The USB issue is worked around by the patch at
> https://lists.linux-foundation.org/pipermail/iommu/2009-August/001657.html  

Nice, thanks for that, this means that your current iommu tree should "just work" on this box, right? (As Greg has acked the patch I assume it is in your tree already).

Anyway going to test this later today or tomorrow.

I have a similar board ASUS p6t7 ws and I'm seeing pretty much the same problems.  I'm tryiing to enable the Vt-d support, and as above, to get the system to boot I must disable legacy USB, and then the system hangs in waiting for uevents.  I tried the iommu=pt, but that doesn't help, but disabling the sound device in the BIOS gets the system booting, so clearly it's some interaction with the intel snd_hda device that is triggering the problem.  I'm running on a 2.6.30-r5 kernel, gentoo distro.  Thanks for the work so far, as you've help me get further along.  Now if I can just integrate the patches, maybe I can get to the point of actually testing Vt-d on this board.

Just some additional data.  I've gotten and built the 2.6.31-rc6 kernel, rebuilt the Nvidia driver, and booted with Vt-d enabled on the board.  As before I disabled legacy USB, and the intel snd-hda device in the BIOS.  I get the following errors which could either be a bug in the Nvidia driver or an issue with DMAR

DRHD: handling fault status reg 202
DMAR:[DMA READ] Request device [07:00.0] fault addr 0
DMAR[fault reason 06] PTE Read access not set

< repeats with different status registers, over and over again, as long as Xserver is active >

Now the graphics card is still working, and 3d acceleration is working as well (i.e. glxgears runs with an appropriate frame rate) albeit slowed by the printk's but obviously somethings wrong.

If I can help test any patches or updates just let me know.

Hmm, it's not clear to me what the current status is with this one - e.g. it's filed against Fedora 10, but people are talking about 2.6.30 and 2.6.31. Anyone care to summarise?

(In reply to comment #43)
> Hmm, it's not clear to me what the current status is with this one - e.g. it's
> filed against Fedora 10, but people are talking about 2.6.30 and 2.6.31. Anyone
> care to summarise?  

It initially happened on F10, but fixing it was a longer process the fixes/workarounds should be in David's IOMMU tree know, dunno if all of them made it into .31.

David?

The Asus isoch bug (all DMA goes south when you load the sound driver) has been root-caused in the lab this week, and I'm putting a patch together now.

There are some other patches which went into .31, some more about to go into .32.

Oh, and the root cause is a BIOS bug. Again.

Yay for closed-source BIOSes.

The workaround for the other BIOS bug which killed this Asus board is queued for 2.6.32, but I've already put it into the rawhide kernel:
https://lists.linux-foundation.org/pipermail/iommu/2009-August/001657.html

OK, sorry for the delay. This should fix it by bypassing the affected IOMMU unit, just for the sound device:
http://git.infradead.org/iommu-2.6.git/commitdiff/e0fc7e0b4

We've _also_ poked Asus and they have a new BIOS available which fixes the problem.

*** Bug 522668 has been marked as a duplicate of this bug. ***

Rahul nominated one of the dupes as f12blocker, so nominating this instead. See all the dupes on 522668: we have a lot of people running into this :(

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

so, here's the scoop on this.

it breaks USB functionality entirely, and in one case at least completely stops the system booting, on several Intel chipset-based motherboards. (It's actually really caused by buggy BIOSes, but we can't sell that to the users). We have at least five reports so far, there's probably more I didn't catch and dupe yet.

If we ship with this, we will have people with unusable systems. The workaround is simple: intel_iommu=off kernel parameter. But it relies on them finding the documentation.

If we disable it by default, the impact is that it breaks PCI passthrough for KVMs. Kyle is almost positive it can't possibly break anything else. The converse workaround would be possible for any virt users who need that to work: intel_iommu=1 .

This is fairly sucky, and late to catch. Honestly it's something I'd like to fix. The above is a summary of what we know so far, I'll do some more research later today (inc. looking for more dupes and check what other distros have done), and possibly poke Jesse's cell.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

also note #530340: it's not strictly a dupe of this issue, but it would be addressed by the fix Kyle would put in place for this issue if we decide to fix it. So that's another report to consider.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

also see 522201 for another system known to throw its toys out of the pram with intel_iommu=1 - smooge, can you check if that system fails with f12's default kernel config?

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

also 495603 - from an earlier kernel when the reporter caught the "DMAR reported at address zero!" message when it was just a warning, I bet that one would fail with f12 kernel. Gustavo, if you read this soon, could you check?

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

What is diffrence between intel_iommu=off and iommu=soft. 
iommu=soft works for me great, but when I used intel_iommu=off after some time my wifi was dead and router need restart (maybe this is ruter problem, not iommu, im not sure). In messages I found:

Nov  7 22:00:19 localhost NetworkManager: <info>  Activation (wlan0) Stage 5 of 5 (IP Configure Commit) complete.
Nov  7 22:00:19 localhost ntpd[1557]: Listening on interface #5 wlan0, 192.168.1.100#123 Enabled
Nov  7 22:01:44 localhost abrtd: Hmm, stray update_client: 'Tworzenie raportów awarii "kernel oops"'
Nov  7 22:01:44 localhost abrt: Kerneloops: Reported 1 kernel oopses to Abrt
Nov  7 22:01:44 localhost abrtd: Directory 'kerneloops-1257627704-1' creation detected
Nov  7 22:01:44 localhost abrtd: Uzyskiwanie lokalnego uniwersalnego, unikalnego identyfikatora
Nov  7 22:01:44 localhost abrtd: New crash, saving...
Nov  7 22:04:26 localhost NetworkManager: <info>  (wlan0): device state change: 8 -> 3 (reason 39)
Nov  7 22:04:26 localhost NetworkManager: <info>  (wlan0): deactivating device (reason: 39).

With default kernel option usb is broken but WIFI and abrtd is OK.

524808 also is related to this topic and contains several people reporting the common 'USB doesn't work' manifestation, don't know if kernel devs would consider it a dupe or not.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

mariusz: I believe intel_iommu=off disables it entirely but iommu=soft falls back to a software implementation. Chuck says that iommu=soft is the preferred workaround, which matches your case.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

David Woodhouse says the correct master bug for the 'my USB no worky' issues is in fact 524808. Will shift the dupe notifications and the F12Blocker status to that bug. Please continue discussion there. Sorry for the mix-up.

He says this bug should be closed, but I'll leave that to him.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

This bug, which causes a machine to crash when the sound driver is loaded, is fixed.