Bug 643901

Summary: KVM PCI device assignment has broken PCI config space emulation
Product: Red Hat Enterprise Linux 6 Reporter: Larry Troan <ltroan>
Component: qemu-kvmAssignee: Don Dutile (Red Hat) <ddutile>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: bhavna.sarathy, chayang, chrisw, ichute, jon.mason, lihuang, llim, ltroan, mkenneth, ovirt-maint, rdoty, rsibley, tburke, virt-maint, yeylon, ykaul
Target Milestone: rc   
Target Release: 6.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 544366 Environment:
Last Closed: 2011-01-22 04:15:04 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 544366    
Bug Blocks: 580566, 580954, 624824    

Comment 1 Larry Troan 2010-10-18 13:33:48 UTC 
+++ This bug was initially created as a clone of Bug #544366 +++

Description of problem:

When assigning a PCI device to a guest the device dependent PCI config space
emulation is broken.  KVM will hardcode offsets to PCI capabilites for MSI and
MSI-X.  This may conflict with existing capabilities or device registers
causing the guest OS device driver to fail.

Version-Release number of selected component (if applicable):

>= kvm-83-105.el5

How reproducible:

This is 100% reproducible for devices that place capabilities or registers in
the area where KVM puts MSI and MSI-X capabilities.  So reproducible, but
device dependent.

Steps to Reproduce:
1. launch guest with assigned PCI device
2. start device driver in guest
3. driver fails to initialize


--- Additional comment from rsibley on 2010-03-16 10:06:28 EDT ---

I have upgraded to the BIOS that AMD says supports IOMMU.

There is still an issue with 10G Neterion Inc. X3100 Series cards, with the
following errors:
Mar 16 09:47:42 perf31 kernel: PCI: Failed to allocate mem resource
#12:8000000@d0000000 for 0000:03:00.0
Mar 16 09:47:42 perf31 kernel: vxge 0000:03:00.0: not enough MMIO resources for
SR-IOV

This is known good module, SR-IOV works on an Intel able to create VFs.

The 1G KUWALA cards still are functional and able create VFs.

/var/log/messages:

Mar 16 09:47:12 perf31 kernel: AMD IOMMU: Using protection domain 24 for device
03:00.0
Mar 16 09:47:12 perf31 kernel: ACPI: PCI interrupt for device 0000:03:00.0
disabled
Mar 16 09:47:42 perf31 kernel: vxge: Copyright(c) 2002-2009 Neterion Inc
Mar 16 09:47:42 perf31 kernel: vxge: Driver version: 2.0.6.18937-k
Mar 16 09:47:42 perf31 kernel: PCI: Enabling device 0000:03:00.0 (0140 -> 0142)
Mar 16 09:47:42 perf31 kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> Link
[LN32] -> GSI 32 (level, high) -> IRQ 138
Mar 16 09:47:42 perf31 kernel: PCI: Failed to allocate mem resource
#12:8000000@d0000000 for 0000:03:00.0
Mar 16 09:47:42 perf31 kernel: vxge 0000:03:00.0: not enough MMIO resources for
SR-IOV
Mar 16 09:47:42 perf31 kernel: eth0: SERIAL NUMBER: SXC0949004
Mar 16 09:47:42 perf31 kernel: eth0: PART NUMBER: X3110SR0001
Mar 16 09:47:42 perf31 kernel: eth0: Neterion X3110 Single-Port SR 10GbE Server
Adapter
Mar 16 09:47:42 perf31 kernel: eth0: MAC ADDR: 00:0C:FC:01:00:C7
Mar 16 09:47:42 perf31 kernel: eth0: Link Width x8
Mar 16 09:47:42 perf31 kernel: eth0: Firmware version : 1.4.4 Date : 09/17/2009
Mar 16 09:47:42 perf31 kernel: eth0: Single Root IOV Mode Enabled
Mar 16 09:47:42 perf31 kernel: eth0: 1 Vpath(s) opened
Mar 16 09:47:42 perf31 kernel: eth0: Interrupt type MSI-X
Mar 16 09:47:42 perf31 kernel: eth0: RTH steering enabled for TCP_IPV4
Mar 16 09:47:42 perf31 kernel: eth0: Tx port steering enabled
Mar 16 09:47:42 perf31 kernel: eth0: Generic receive offload enabled
Mar 16 09:47:42 perf31 kernel: eth0: Rx doorbell mode enabled
Mar 16 09:47:42 perf31 kernel: eth0: VLAN tag stripping Enabled
Mar 16 09:47:42 perf31 kernel: eth0: Ring blocks : 2
Mar 16 09:47:42 perf31 kernel: eth0: Fifo blocks : 14
Mar 16 09:47:42 perf31 kernel: eth0: MTU is 1500


./lspci
03:00.0 Class 0200: Device 17d5:5833 (rev 02)
 Subsystem: Device 17d5:6030
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping-
SERR+ FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
 Latency: 0, Cache Line Size: 32 bytes
 Interrupt: pin A routed to IRQ 138
 Region 0: Memory at ce800000 (64-bit, prefetchable) [size=8M]
 Region 2: Memory at ce201000 (64-bit, prefetchable) [size=4K]
 Region 4: Memory at ce200000 (64-bit, prefetchable) [size=256]
 [virtual] Expansion ROM at ce280000 [disabled] [size=512K]
 Capabilities: [40] Power Management version 3
  Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot-,D3cold-)
  Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [70] Express (v1) Endpoint, MSI 00
  DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <2us, L1 <2us
   ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
  DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
   RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
   MaxPayload 128 bytes, MaxReadReq 512 bytes
  DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
  LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency L0 <256ns, L1
<4us
   ClockPM- Surprise- LLActRep- BwNot-
  LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
   ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
  LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt-
ABWMgmt-
 Capabilities: [a0] MSI-X: Enable+ Count=4 Masked-
  Vector table: BAR=2 offset=00000000
  PBA: BAR=2 offset=00000800
 Capabilities: [c0] Vital Product Data
  Not readable
 Capabilities: [50] MSI: Enable- Count=1/4 Maskable+ 64bit+
  Address: 0000000000000000  Data: 0000
  Masking: 00000000  Pending: 00000000
 Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
  ARICap: MFVC- ACS-, Next Function: 0
  ARICtl: MFVC- ACS-, Function Group: 0
 Capabilities: [110] Advanced Error Reporting
  UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-
  UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-
  UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol-
  CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
  AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
 Capabilities: [150] Power Budgeting <?>
 Capabilities: [160] Device Serial Number 00-0c-fc-00-00-01-00-c7
 Capabilities: [170] Single Root I/O Virtualization (SR-IOV)
  IOVCap: Migration-, Interrupt Message Number: 000
  IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
  IOVSta: Migration-
  Initial VFs: 16, Total VFs: 16, Number of VFs: 1, Function Dependency Link:
00
  VF offset: 1, stride: 1, Device ID: 5833
  Supported Page Size: 000007ff, System Page Size: 00000001
  Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
  Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
  Region 4: Memory at 0000000000000000 (64-bit, prefetchable)
  VF Migration: offset: 00000000, BIR: 0
 Capabilities: [1b0] Vendor Specific Information <?>
 Kernel driver in use: vxge
 Kernel modules: vxge



cat /proc/iomem
00010000-0009d7ff : System RAM
0009d800-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cafff : Video ROM
000cb000-000cbfff : Adapter ROM
000d1800-000d19ff : Adapter ROM
000f0000-000fffff : System ROM
00100000-c7e7ffff : System RAM
  00200000-00484429 : Kernel code
  0048442a-005c9b8f : Kernel data
  08000000-0bffffff : GART
c7e80000-c7e8afff : ACPI Tables
c7e8b000-c7e8cfff : ACPI Non-volatile Storage
c7e8d000-c7ffffff : reserved
c8004000-c8007fff : amd_iommu
c8024000-c80243ff : 0000:00:11.0
  c8024000-c80243ff : ahci
c8024400-c80244ff : 0000:00:12.2
  c8024400-c80244ff : ehci_hcd
c8024800-c80248ff : 0000:00:13.2
  c8024800-c80248ff : ehci_hcd
c8025000-c8025fff : 0000:00:12.0
  c8025000-c8025fff : ohci_hcd
c8026000-c8026fff : 0000:00:12.1
  c8026000-c8026fff : ohci_hcd
c8027000-c8027fff : 0000:00:13.0
  c8027000-c8027fff : ohci_hcd
c8028000-c8028fff : 0000:00:13.1
  c8028000-c8028fff : ohci_hcd
c8029000-c8029fff : 0000:00:14.5
  c8029000-c8029fff : ohci_hcd
c8100000-c8bfffff : PCI Bus #01
  c8100000-c8103fff : 0000:01:00.0
    c8100000-c8103fff : igb
  c8104000-c8107fff : 0000:01:00.1
    c8104000-c8107fff : igb
  c8120000-c813ffff : 0000:01:00.0
    c8120000-c813ffff : igb
  c8140000-c815ffff : 0000:01:00.1
    c8140000-c815ffff : igb
  c8160000-c817ffff : 0000:01:00.0
    c8160000-c8163fff : 0000:01:10.0
      c8160000-c8163fff : kvm_assigned_device
    c8164000-c8167fff : 0000:01:10.2
      c8164000-c8167fff : igbvf
  c8180000-c819ffff : 0000:01:00.0
    c8180000-c8183fff : 0000:01:10.0
      c8180000-c8183fff : kvm_assigned_device
    c8184000-c8187fff : 0000:01:10.2
      c8184000-c8187fff : igbvf
  c81a0000-c81bffff : 0000:01:00.1
    c81a0000-c81a3fff : 0000:01:10.1
      c81a0000-c81a3fff : igbvf
    c81a4000-c81a7fff : 0000:01:10.3
      c81a4000-c81a7fff : igbvf
  c81c0000-c81dffff : 0000:01:00.1
    c81c0000-c81c3fff : 0000:01:10.1
      c81c0000-c81c3fff : igbvf
    c81c4000-c81c7fff : 0000:01:10.3
      c81c4000-c81c7fff : igbvf
  c8400000-c87fffff : 0000:01:00.0
    c8400000-c87fffff : igb
  c8800000-c8bfffff : 0000:01:00.1
    c8800000-c8bfffff : igb
ca000000-cdffffff : PCI Bus #02
  ca000000-cbffffff : 0000:02:00.0
    ca000000-cbffffff : bnx2
  cc000000-cdffffff : 0000:02:00.1
    cc000000-cdffffff : bnx2
ce000000-ce0fffff : PCI Bus #04
  ce000000-ce003fff : 0000:04:00.0
    ce000000-ce003fff : qla2xxx
  ce004000-ce007fff : 0000:04:00.1
    ce004000-ce007fff : qla2xxx
ce100000-ce1fffff : PCI Bus #05
  ce100000-ce10ffff : 0000:05:06.0
  ce120000-ce13ffff : 0000:05:06.0
ce200000-ceffffff : PCI Bus #03
  ce200000-ce2000ff : 0000:03:00.0
    ce200000-ce2000ff : vxge
  ce201000-ce201fff : 0000:03:00.0
    ce201000-ce201fff : vxge
  ce280000-ce2fffff : 0000:03:00.0
  ce800000-ceffffff : 0000:03:00.0
    ce800000-ceffffff : vxge
cf000000-cf7fffff : PCI Bus #01
  cf000000-cf3fffff : 0000:01:00.0
  cf400000-cf7fffff : 0000:01:00.1
cf800000-cf8fffff : PCI Bus #02
  cf800000-cf81ffff : 0000:02:00.0
  cf820000-cf83ffff : 0000:02:00.1
cf900000-cf9fffff : PCI Bus #03
cfa00000-cfafffff : PCI Bus #04
  cfa00000-cfa3ffff : 0000:04:00.0
  cfa40000-cfa7ffff : 0000:04:00.1
d0000000-d7ffffff : PCI Bus #05
  d0000000-d7ffffff : 0000:05:06.0
e0000000-efffffff : reserved
fec00000-fec0ffff : reserved
fee00000-fee00fff : reserved
fff00000-ffffffff : reserved
100000000-837ffffff : System RAM


BIOS version:

# dmidecode 2.10
SMBIOS 2.5 present.
56 structures occupying 1852 bytes.
Table at 0xC7EDA000.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
 Vendor: Phoenix Technologies Ltd.
 Version: PDNAX1-A
 Release Date: 01/26/2010
 Address: 0xE3230
 Runtime Size: 118224 bytes


2.6.18-191.el5 #1 SMP Mon Mar 1 15:59:02 EST 2010 x86_64 x86_64 x86_64
GNU/Linux

kmod-kvm-83-160.el5
kvm-83-160.el5
etherboot-zroms-kvm-5.4.4-13.el5
kvm-qemu-img-83-160.el5

bob

--- Additional comment from ltroan on 2010-10-18 09:12:43 EDT ---

Will clone this bug for RHEL6.... No bug open at this time but Exar (formerly
Neterion) indicating failure on the latest RHEL6 build.