Description of problem: /var/log/messages fills all disk space with error messages from DMAR. HW is Lenovo T410 with Intel CORE i5 cpu. How reproducible: 100% failure when feature activated in BIOS before booting Fedora. No failure when feature deactivated in BIOS before booting Fedora. Steps to Reproduce: 1. Power off system 2. Power on system and enter BIOS setup 3. From CPU option, enable VT-d feature 4. Save BIOS settings & boot system Actual results: Error messages flood /var/log/messages (1+ set of messages follows): Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02] Pce [0d:00.0] fault addr fce [0d:00.0] fault addr fffDMAR:[fault reace [0d:00.0] fault addr ce [0d:00.0] fault addr fffffDMAR:[fault reasce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02] Prce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reasonce [0d:00.0] fault addr DMAR:[fault reasce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02]ce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02] Present bice [0d:00.0] fault adce [0d:00.0] fault addr fffff000 DMAR:[fault reasce [0d:00.0] fault addr fffff000DMAR:[fault rece [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02] Present bit in contexce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMARce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 0ce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reason 02] Prce [0d:00.0] fault addr fffff000 Jun 19 00:38:11 PLS-T410 kernel: DMAR:[fault reasonce [0d:00.0] fault addr fffff000 Expected results: No error messages from DMAR Additional info: from messages file: Linux version 2.6.33.3-85.fc13.x86_64 (mockbuild.fedoraproject.org) (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #1 SMP Thu May 6 18:09:49 UTC 2010 kernel: CPU0: Intel(R) Core(TM) i5 CPU M 520 @ 2.40GHz stepping 02 kernel: Booting Node 0, Processors #1 kernel: CPU 1 MCA banks SHD:2 SHD:3 SHD:5 SHD:6 kernel: #2 kernel: CPU 2 MCA banks CMCI:2 CMCI:3 CMCI:5 SHD:6 kernel: #3 Ok. kernel: CPU 3 MCA banks SHD:2 SHD:3 SHD:5 SHD:6 kernel: Brought up 4 CPUs kernel: Total of 4 processors activated (19150.66 BogoMIPS). Temporary resolution: 1. Start system from power off state and enter BIOS setup 2. Deactivate VT-d feature 3. Save configuration and reboot.
(In reply to comment #1) > Linux version 2.6.33.3-85.fc13.x86_64 (mockbuild.fedoraproject.org) > (gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) ) #1 SMP Thu May 6 18:09:49 > UTC 2010 > There have been three kernel updates since that version. Did you try any of them before reporting this problem?
I have the same problem here on 2.6.33.5-124.fc13.i686.PAE.
Having the same problem on a ThinkPad T410 with kernel-2.6.33.5-124.fc13.x86_64 (latest bios 1.25) This is the PCI device in question # lspci -vvnn -s 0d:00.0 0d:00.0 SD Host controller [0805]: Ricoh Co Ltd Device [1180:e822] (rev 01) Subsystem: Lenovo Device [17aa:2133] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at f2500000 (32-bit, non-prefetchable) [size=256] Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [78] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=2 PME- Capabilities: [80] Express (v1) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn+ AttnInd+ PwrInd+ RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntrySize=0 Arb: Fixed- WRR32- WRR64- WRR128- 100ns- - - onfig- TableOffset=0 Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Fixed- RR32- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [800 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: sdhci-pci Kernel modules: sdhci-pci
I'm working on a quirk to handle this buggy device.
*** Bug 587178 has been marked as a duplicate of this bug. ***
This happens on at least two different kinds of Ricoh multifunction devices. In one case device 00.0 is a cardbus bridge and in the other case it's shown as an SD controller: 0d:00.0 SD Host controller [0805]: Ricoh Co Ltd Device [1180:e822] (rev 01) 0d:00.1 System peripheral [0880]: Ricoh Co Ltd Device [1180:e230] (rev 01) 0d:00.3 FireWire (IEEE 1394) [0c00]: Ricoh Co Ltd Device [1180:e832] (rev 01) 04:00.0 CardBus bridge [0607]: Ricoh Co Ltd Device [1180:e476] (rev 02) 04:00.4 FireWire (IEEE 1394) [0c00]: Ricoh Co Ltd Device [1180:e832] (rev 03) (prog-if 10)
Just in case anybody else is in a hurry and needs a workaround, here's a patch (against 2.6.35-rc5) that kludges all the DMAR mappings to point to the first device. Tested with Cardbus (a USB controller) and Firewire, but don't be surprised if it glues your cat to the carpet or something. --- a/drivers/pci/intel-iommu.c 2010-07-13 07:55:33.000000000 +1000 +++ b/drivers/pci/intel-iommu.c 2010-08-03 22:19:09.000000000 +1000 @@ -2560,10 +2560,12 @@ return 0; } +struct pci_dev *ricohdev = 0; + static dma_addr_t __intel_map_single(struct device *hwdev, phys_addr_t paddr, size_t size, int dir, u64 dma_mask) { - struct pci_dev *pdev = to_pci_dev(hwdev); + struct pci_dev *tmp, *pdev = to_pci_dev(hwdev); struct dmar_domain *domain; phys_addr_t start_paddr; struct iova *iova; @@ -2574,6 +2576,17 @@ BUG_ON(dir == DMA_NONE); + tmp = (pdev->vendor==0x1180) ? pdev : pdev->bus->self; + if (tmp && tmp->vendor==0x1180 && + (tmp->device==0xe822 || + tmp->device==0xe230 || + tmp->device==0xe832 || + tmp->device==0xe476)) { + if (!ricohdev) + ricohdev = pci_get_domain_bus_and_slot(0, tmp->bus->number, tmp->devfn & ~7); + pdev = ricohdev; + } + if (iommu_no_mapping(hwdev)) return paddr; @@ -2716,7 +2729,7 @@ size_t size, enum dma_data_direction dir, struct dma_attrs *attrs) { - struct pci_dev *pdev = to_pci_dev(dev); + struct pci_dev *tmp, *pdev = to_pci_dev(dev); struct dmar_domain *domain; unsigned long start_pfn, last_pfn; struct iova *iova; @@ -2724,6 +2737,15 @@ if (iommu_no_mapping(dev)) return; + + tmp = (pdev->vendor==0x1180) ? pdev : pdev->bus->self; + if (tmp && tmp->vendor==0x1180 && + (tmp->device==0xe822 || + tmp->device==0xe230 || + tmp->device==0xe832 || + tmp->device==0xe476)) { + pdev = ricohdev; + } domain = find_domain(pdev); BUG_ON(!domain);
ThinkPad T510 w/ i7 (common devel box) also has this device and issue.
Just verified that this problem still exists on my T510 with 2.6.34.6-47.fc13.x86_64.
This issue occurs on my T410 with Core i5 540m.
Also a problem on my ThinkPad T510 with i7 processor.
This is a hardware bug, and there's not really any good way to work around it.
The 'workaround' for now is to disable DMAR by either disabling VT-d in your BIOS, or by booting with intel_iommu=off. Thanks, Kyle
After disabling Firewire in the BIOS, the errors ceased. I left the memory card reader on, but don't have a card to test it at the moment.
(In reply to comment #14) > After disabling Firewire in the BIOS, the errors ceased. I left the memory > card reader on, but don't have a card to test it at the moment. Where did you find the option to disable Firewire? I just tried to do this on my T510 (BIOS 1.27), and I can't find anywhere to turn it off.
I am running BIOS 1.24 according to dmidecode, and it was located in the Security Section under I/O Port Access. I just checked that with my older T61 and the setting seems to be in the same place.
(In reply to comment #16) > I am running BIOS 1.24 according to dmidecode, and it was located in the > Security Section under I/O Port Access. I just checked that with my older T61 > and the setting seems to be in the same place. Because obviously, that should be under Security, not Config. :-( On the bright side, I'm seeing the same thing. Once IEEE1394 is disabled, I can run with VT-d enabled without any error messages in the log.
*** Bug 634135 has been marked as a duplicate of this bug. ***
This bug is severe enough to cause hard lockups under some conditions. David, did you get anywhere on writing a quirk? My patch does what's needed, but feels like an ugly patch on a critical path...
Same issue here, 2.6.34.6-54.fc13.x86_64 on a brand new Lenovo Thinkpad T510+.
*** Bug 635678 has been marked as a duplicate of this bug. ***
I also got this problem on T410, almost given a day to understand that problem is in kernel after trying different installations (F13 and F14 live/DVD) I used intel_iommu=off. Now it feels so good to see the real dmesg output after starting my laptop, else it used to show full of those 3 debug lines repeatedly. If this is not cloned for Fedora 14 then we should have new bug and marked as F14Target.
We had this problem with a number of T510s in the office. As well as the issue with flooding /var/log/messages it can also manifest during the installation phase where disks are being fscked. On investigation we have found, that on these machines at least, turning off VT-d in the BIOS corrected the problem. The other virtualization related options in the BIOS can however be left on with no apparent ill effects.
kernel-2.6.35.6-45.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/kernel-2.6.35.6-45.fc14
kernel-2.6.34.7-61.fc13 has been submitted as an update for Fedora 13. https://admin.fedoraproject.org/updates/kernel-2.6.34.7-61.fc13
I installed kernel-2.6.34.7-61.fc13 on my T410, and enabled VT-d in the BIOS. With this, I am no longer seeing the DMAR faults. Will this fix also be pushed for RHEL6?
kernel-2.6.35.6-45.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.
kernel-2.6.34.7-61.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.
Will this fix also be pushed for RHEL6?
Upgraded firmware on mellanox infiniband connectx hardware and got following spam in system log. [ 1680.962538] DRHD: handling fault status reg 302 [ 1680.967048] DMAR:[DMA Read] Request device [04:00.6] fault addr f647a000 [ 1680.967049] DMAR:[fault reason 02] Present bit in context entry is clear This also broke infiniband on f13/f14 machines. I used the iommu workaround described above, and infiniband now works. RHEL6.0 doesn't exhibit this problem with the updated firmware: Linux mrg-03.mpc.lab.eng.bos.redhat.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue Mx Is this a kernel.org regression? lspci results: [root@mrg-04 ~]# lspci 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Po) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Po) 00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 ) 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 ) 00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 ) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Po) 00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Po) 00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Register) 00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Regis) 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Reg) 00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Control) 00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Control) 00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Contro) 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Control) 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Control) 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Contro) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller () 00:1f.2 IDE interface: Intel Corporation 82801IB (ICH9) 2 port SATA IDE Control) 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit ) 01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit ) 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit ) 02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit ) 03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 0) 04:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s ) 08:03.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (re) fe:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture) fe:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture) fe:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) fe:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) fe:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05) fe:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05) fe:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) fe:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture) ff:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture) ff:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) ff:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) ff:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05) ff:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05) ff:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) ff:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Cont) [root@mrg-04 ~]#
This message is a reminder that Fedora 13 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 13. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '13'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 13's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 13 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
(In reply to comment #30) > Upgraded firmware on mellanox infiniband connectx hardware and got following > spam in system log. > > [ 1680.962538] DRHD: handling fault status reg 302 > [ 1680.967048] DMAR:[DMA Read] Request device [04:00.6] fault addr f647a000 > [ 1680.967049] DMAR:[fault reason 02] Present bit in context entry is clear > This has absolutely nothing to do with workarounds needed for buggy Ricoh multifunction devices. Please open a new bug...