This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1306105 - booting VM with a SR-IOV VF assigned fails with Bad Address
booting VM with a SR-IOV VF assigned fails with Bad Address
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: qemu (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Fedora Virtualization Maintainers
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-09 21:28 EST by Joshua Roys
Modified: 2016-05-02 16:36 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-02 16:36:32 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg (81.84 KB, text/plain)
2016-02-09 21:28 EST, Joshua Roys
no flags Details
lspci -vvvnnxxx (70.10 KB, text/plain)
2016-03-11 06:56 EST, Joshua Roys
no flags Details

  None (edit)
Description Joshua Roys 2016-02-09 21:28:28 EST
Created attachment 1122605 [details]
dmesg

Description of problem:
I'm fighting an uphill battle to get SR-IOV working with the HW I have currently.  After learning that Xeon E3-1200's don't support ACS but that the PCH does, I moved my i350 NIC to a PCH PCIe port and got correct IOMMU groups.  I have to boot with pci=assign-busses,realloc because Supermicro assumes you won't get this far.  Now, finally, qemu dies with the following:

[...] /usr/bin/qemu-kvm [...] -device vfio-pci,host=04:13.0,id=hostdev0,bus=pci.0,addr=0x3 [...]
2016-02-10T02:14:16.361029Z qemu-system-x86_64: VFIO_MAP_DMA: -14
2016-02-10T02:14:16.361061Z qemu-system-x86_64: vfio_dma_map(0x5639e63a8190, 0xfc050000, 0x4000, 0x7f4bbc6d1000) = -14 (Bad address)
qemu: hardware error: vfio: DMA mapping failed, unable to continue
CPU #0:
EAX=00000103 EBX=00001800 ECX=00000103 EDX=00000cfc
ESI=00000000 EDI=00000000 EBP=00000007 ESP=00006d7c
EIP=7ffaf546 EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6b70 00000037
IDT=     000f6bae 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
CPU #1:
inputs_connect: inputs channel client create
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306c1
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

Is there any hope?  I'm not sure what is causing this latest roadblock.


Version-Release number of selected component (if applicable):
kernel-4.3.4-200.fc22.x86_64
qemu-kvm-2.3.1-11.fc22.x86_64


How reproducible:
Supermicro X10SAE with an Intel i350-T2 in the top x1 PCIe port


Steps to Reproduce:
1. procure an X10SAE or equivalent, E3-1226v3 and i350-T2
2. boot with intel_iommu=on pci=assign-busses,realloc
3. create VFs via echo to /sys/.../sriov_numvfs
4. assign VF to VM
5. attempt to start VM


Actual results:
Splat as above.


Expected results:
Profit.


Additional info:
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller
           +-02.0  Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller
           +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
           +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
           +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
           +-1c.0-[01]--
           +-1c.3-[02]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.4-[03-04]--+-[0000:04]-+-10.0  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           +-10.4  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           +-11.0  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           +-11.4  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           +-12.0  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           +-12.4  Intel Corporation I350 Ethernet Controller Virtual Function
           |               |           \-13.0  Intel Corporation I350 Ethernet Controller Virtual Function
           |               \-[0000:03]-+-00.0  Intel Corporation I350 Gigabit Network Connection
           |                           \-00.1  Intel Corporation I350 Gigabit Network Connection
           +-1c.5-[05-06]----00.0-[06]----03.0  Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
           +-1c.6-[07]----00.0  Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
           +-1c.7-[08]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1
           +-1f.0  Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
           +-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller
           \-1f.6  Intel Corporation 8 Series Chipset Family Thermal Management Controller

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
Comment 1 Cole Robinson 2016-02-10 14:32:38 EST
For starters I'd suggest updating to fedora 23 since there's a lot of moving parts here.

Is this a manual qemu-kvm command line, or via libvirt?

Alex, any thoughts here?
Comment 2 Joshua Roys 2016-02-10 14:50:20 EST
I'll see about upgrading.  This is via libvirt.
Comment 3 Alex Williamson 2016-02-10 15:03:35 EST
Are you trying to boot a VM with more memory than is available on the host?  There's no overcommit with device assignment.
Comment 4 Joshua Roys 2016-02-10 15:30:13 EST
No, IIRC I was booting a VM with 2GB, 32GB on the host.
Comment 5 Joshua Roys 2016-02-18 23:09:09 EST
kernel-4.3.5-300.fc23.x86_64
qemu-kvm-2.4.1-6.fc23.x86_64

2016-02-19 04:02:27.876+0000: starting up libvirt version: 1.2.18.2, package: 2.fc23 (Fedora Project, 2016-01-21-00:54:13, buildhw-01-nfs.phx2.fedoraproject.org), qemu version: 2.4.1 (qemu-2.4.1-6.fc23)
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/bin/qemu-kvm -name ipa01 -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu Haswell-noTSX -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 2eefb093-fef6-4fa0-a719-d7589089b47e -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/ipa01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/mnt/cumulonimbus/vm/ipa01.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fd:10:8a,bus=pci.0,addr=0x9 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=04:10.0,id=hostdev0,bus=pci.0,addr=0x3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 0.727000 ms, bitrate 37925925925 bps (36168.981481 Mbps)
red_dispatcher_set_cursor_peer: 
inputs_connect: inputs channel client create
2016-02-19T04:02:28.747947Z qemu-system-x86_64: VFIO_MAP_DMA: -14
2016-02-19T04:02:28.747964Z qemu-system-x86_64: vfio_dma_map(0x5564f73d3450, 0xfc050000, 0x4000, 0x7f90d7fcc000) = -14 (Bad address)
qemu: hardware error: vfio: DMA mapping failed, unable to continue
CPU #0:
EAX=00000103 EBX=00000000 ECX=00000103 EDX=00000cfc
ESI=00000018 EDI=00000000 EBP=000f5225 ESP=00006ecc
EIP=7ffaf431 EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6ac0 00000037
IDT=     000f6afe 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
CPU #1:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000306c1
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
2016-02-19 04:02:30.180+0000: shutting down
Comment 6 Joshua Roys 2016-03-09 21:34:27 EST
No change under 4.4.3-300.fc23.x86_64
Comment 7 Alex Williamson 2016-03-09 22:35:18 EST
Is the problem limited to the VFs in this system?  Can you assign the I210?  Are the I350 PFs assignable, with or without pci=assign-busses,realloc?

Please provide 'sudo lspci -vvvs 00:1c.4'.  Based on the addresses of the VFs, I'm guessing that root port does not support ARI, which may be the key to reproducing.
Comment 8 Alex Williamson 2016-03-10 12:53:38 EST
I've got a fairly similar system, 6-series chipset vs 8-series, 82576 vs I350:

# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GM107 [GeForce GTX 750]
           |            \-00.1  NVIDIA Corporation Device 0fbc
           +-02.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
           +-16.0  Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller
           +-1c.0-[02-03]--+-[0000:03]-+-10.0  Intel Corporation 82576 Virtual Function
           |               |           +-10.2  Intel Corporation 82576 Virtual Function
           |               |           +-10.4  Intel Corporation 82576 Virtual Function
           |               |           +-10.6  Intel Corporation 82576 Virtual Function
           |               |           +-11.0  Intel Corporation 82576 Virtual Function
           |               |           +-11.2  Intel Corporation 82576 Virtual Function
           |               |           \-11.4  Intel Corporation 82576 Virtual Function
           |               \-[0000:02]-+-00.0  Intel Corporation 82576 Gigabit Network Connection
           |                           \-00.1  Intel Corporation 82576 Gigabit Network Connection
           +-1c.5-[04]----00.0  ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
           +-1c.6-[05]----00.0  Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
           +-1c.7-[06-07]----00.0-[07]--
           +-1d.0  Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation H67 Express Chipset Family LPC Controller
           +-1f.2  Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller
           \-1f.3  Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller

pci=assign-busses is necessary to enable SR-IOV due to the lack of ARI support on the PCH root port and lack of BIOS support for enumerating SR-IOV.  'realloc' is not required because the VFs on the 82576 already fit within the minimum PCI bridge aperture, but it doesn't seem to hurt.  I expect this is also true of the I350 since they're effectively the same part.

Running kernel 4.4.3-300.fc23.x86_64, qemu-kvm qemu-2.4.1-7.fc23.x86_64.  So ARI is likely not the issue and I'm running the same kernel and qemu and not seeing this problem.  Have you modified /usr/libvirt/qemu.conf in any way from default?  Are you running virsh as root, ie. the system session, or using virt-manager to connect as root vs a normal user?
Comment 9 Joshua Roys 2016-03-11 06:24:03 EST
Attempting to assign 03:00.0, the first PF, fails with the same error.  I'll try without the pci= arguments next.  I haven't modified qemu.conf since install.  I'm using virt-manager to assign the devices and boot the VM.

00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Bus: primary=00, secondary=03, subordinate=04, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: de400000-de7fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt+
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #4, PowerLimit 10.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable+ ErrNon-Fatal+ ErrFatal+ PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0805]
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: pcieport
        Kernel modules: shpchp
00: 86 80 18 8c 07 00 10 00 d5 00 04 06 10 00 81 00
10: 00 00 00 00 00 00 00 00 00 03 04 00 f0 00 00 20
20: 40 de 70 de f1 ff 01 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 10 00
40: 10 80 42 01 00 80 00 00 07 00 10 00 12 3c 32 05
50: 42 00 12 f0 00 b2 24 00 00 00 40 00 07 00 00 00
60: 00 00 00 00 17 08 00 00 00 04 00 00 00 00 00 00
70: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 05 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 0d a0 00 00 d9 15 05 08 00 00 00 00 00 00 00 00
a0: 01 00 03 c8 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 01 42 18 00 00 08 80 11 8b 00 00 00 00
e0: 02 3f 00 00 00 00 00 00 13 00 00 00 00 00 00 00
f0: 50 00 00 00 40 00 00 00 b1 0f 06 08 00 28 00 05
Comment 10 Joshua Roys 2016-03-11 06:56 EST
Created attachment 1135200 [details]
lspci -vvvnnxxx

I can assign the i350 PF without any pci= kernel arguments.  Attached is a full `lspci -vvvnnxxx` from this boot.
Comment 11 Joshua Roys 2016-03-11 07:07:20 EST
Well, booting with pci=assign-busses allows me to make 7 VFs on the first i350 function.  The second function fails with "not enough MMIO resources for SR-IOV."  However, I can assign and boot one of the 7 VFs!  I don't remember if I had tried pci=realloc first and never just assign-busses by itself or maybe the upgrade to F23 got me half working.  Thank you!

Do you think it is a hopeless cause without Supermicro enabling SR-IOV in their BIOS to get the second i350 function to make VFs?  Or is there a bug in the kernel PCIe realloc bits?
Comment 12 Cole Robinson 2016-05-02 16:36:32 EDT
(In reply to Joshua Roys from comment #11)
> Well, booting with pci=assign-busses allows me to make 7 VFs on the first
> i350 function.  The second function fails with "not enough MMIO resources
> for SR-IOV."  However, I can assign and boot one of the 7 VFs!  I don't
> remember if I had tried pci=realloc first and never just assign-busses by
> itself or maybe the upgrade to F23 got me half working.  Thank you!
> 

Sounds like it's working, so closing

> Do you think it is a hopeless cause without Supermicro enabling SR-IOV in
> their BIOS to get the second i350 function to make VFs?  Or is there a bug
> in the kernel PCIe realloc bits?

Might be better to take additional discussion to https://www.redhat.com/mailman/listinfo/vfio-users

Note You need to log in before you can comment on or make changes to this bug.