Bug 984400

Summary: Intel 82599EB physical nic can't be found in seabios when assigning it to guest by vfio-pci
Product: Red Hat Enterprise Linux 7 Reporter: mazhang <mazhang>
Component: qemu-kvmAssignee: Alex Williamson <alex.williamson>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, chayang, hhuang, juzhang, mazhang, michen, qzhang, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 984790 (view as bug list) Environment:
Last Closed: 2014-02-04 16:09:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 984790    

Description mazhang 2013-07-15 06:52:38 UTC 
Description of problem:
assign Intel 82599EB nic to guest by vfio-pci, boot up guest and try install os from pxe, but can't found nic in seabios.

Version-Release number of selected component (if applicable):

host configuration:
qemu-kvm-1.5.1-2.el7.x86_64
kernel-3.10.0-0.rc7.64.el7.x86_64
seabios-bin-1.7.2.2-2.el7.noarch


How reproducible:
100%


Steps to Reproduce:
1. unbind PF from host
2. bind it to vfio-pci
3. boot up guest after enter seabios press "F12"
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 4G \
-smp 2,sockets=1,cores=2,threads=1,maxcpus=16 \
-enable-kvm \
-name win7 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,clock=host,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:6666,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-vga qxl \
-spice port=5900,disable-ticketing \
-global PIIX4_PM.disable_s3=0 \
-global PIIX4_PM.disable_s4=0 \
-drive file=/home/rhel7.qcow3,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads \
-device virtio-blk-pci,scsi=off,bus=pci.0,drive=drive-virtio-disk0,id=virtio-disk0 \
-device vfio-pci,host=06:00.0,id=pf \

Actual results:
can't found the nic in boot device list

Expected results:
can be found it in boot device list

Additional info:
boot up guest with q35 also hit this problem.
Comment 2 Alex Williamson 2013-07-15 13:10:35 UTC 
Does the device have an option ROM?  lspci -vvv on host
Comment 3 mazhang 2013-07-16 03:41:42 UTC 
1 yes, it has an option ROM.
[root@intel-e5530-8-2 0000:06:00.0]# file rom 
rom: BIOS (ia32) ROM Ext. (124*512)

2 This host has been install rhel6u5,here is the prompt by "lspci -vvv"
[root@intel-e5530-8-2 ~]# lspci -vvv -s 06:00.0
06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 65
	Region 0: Memory at df300000 (64-bit, non-prefetchable) [size=512K]
	Region 2: I/O ports at ecc0 [size=32]
	Region 4: Memory at df2f8000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at df200000 [disabled] [size=512K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00578  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable- Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
		Unknown small resource type 00, will not decode more.
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-c3-d0-3c
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 128, stride: 2, Device ID: 10ed
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000df400000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000df500000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: pci-stub
	Kernel modules: ixgbe

3 rhel6u5 also hit this problem.
Comment 4 Alex Williamson 2013-12-19 21:49:29 UTC 
I cannot reproduce this.  I used your exact commandline with an X540-AT2 Intel 10G NIC and it boots from the network every time.  Please verify that you can boot from this NIC on bare metal and that this bug is still reproducible.

qemu-kvm-1.5.3-30.el7.x86_64
kernel-3.10.0-64.el7.x86_64
seabios-bin-1.7.2.2-6.el7.x86_64
Comment 5 mazhang 2013-12-20 11:53:25 UTC 
I can reproduce this bug with qemu-kvm-1.5.3-30.el7.x86_64

qemu-kvm-1.5.3-30.el7.x86_64
kernel-3.10.0-61.el7.x86_64
seabios-1.7.2.2-6.el7.x86_64
seabios-bin-1.7.2.2-6.el7.x86_64

Start bios (version seabios-1.7.2.2-6.el7)
Ram Size=0xe0000000 (0x0000000020000000 high)
Relocating low data from 0x000e4140 to 0x000ef780 (size 2161)
Relocating init from 0x000e49b1 to 0xdffe2e30 (size 53415)
CPU Mhz=2398
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 7 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: map device bdf=00:03.0  bar 0, addr 0000c000, size 00000040 [io]
PCI: map device bdf=00:02.0  bar 3, addr 0000c040, size 00000020 [io]
PCI: map device bdf=00:01.1  bar 4, addr 0000c060, size 00000010 [io]
PCI: map device bdf=00:02.0  bar 0, addr f4000000, size 04000000 [mem]
PCI: map device bdf=00:02.0  bar 1, addr f8000000, size 04000000 [mem]
PCI: map device bdf=00:02.0  bar 6, addr fc000000, size 00010000 [mem]
PCI: map device bdf=00:04.0  bar 0, addr fc010000, size 00004000 [mem]
PCI: map device bdf=00:04.0  bar 3, addr fc014000, size 00004000 [mem]
PCI: map device bdf=00:02.0  bar 2, addr fc018000, size 00002000 [mem]
PCI: map device bdf=00:03.0  bar 1, addr fc01a000, size 00001000 [mem]
PCI: init bdf=00:00.0 id=8086:1237
PCI: init bdf=00:01.0 id=8086:7000
PIIX3/PIIX4 init: elcr=00 0c
PCI: init bdf=00:01.1 id=8086:7010
PCI: init bdf=00:01.3 id=8086:7113
Using pmtimer, ioport 0xb008, freq 3579 kHz
PCI: init bdf=00:02.0 id=1b36:0100
PCI: init bdf=00:03.0 id=1af4:1001
PCI: init bdf=00:04.0 id=8086:10ed
Found 2 cpu(s) max supported 16 cpu(s)
MP table addr=0x000fded0 MPC table addr=0x000fdee0 size=364
SMBIOS ptr=0x000fdeb0 table=0xdffffbf0 size=1026
Scan for VGA option rom
Running option rom at c000:0003
Start SeaVGABIOS (version seabios-1.7.2.2-6.el7)
enter vga_post:
   a=00000010  b=0000ffff  c=00000000  d=0000ffff ds=0000 es=f000 ss=0000
  si=00000000 di=0000d150 bp=00000000 sp=00006de2 cs=f000 ip=c4dc  f=0000
VBE DISPI: bdf 00:02.0, bar 0
VBE DISPI: lfb_addr=f4000000, size 64 MB
Turning on vga text mode console
set VGA mode 3
SeaBIOS (version seabios-1.7.2.2-6.el7)
Machine UUID feebc8fd-f8b0-4e75-abc3-e63fcdb67170
Found 0 lpt ports
Found 0 serial ports
ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 2 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:3
Searching bootorder for: /pci@i0cf8/*@3
PS2 keyboard initialized
All threads complete.
Scan for option roms
Searching bootorder for: /rom@genroms/kvmvapic.bin

Press F12 for boot menu.

Select boot device:

1. Virtio disk PCI:0:3
2. Legacy option rom

This problem may just happened with 82599EB nic.
Comment 6 Alex Williamson 2013-12-20 18:06:26 UTC 
Can the bare-metal system boot from this NIC?
Comment 7 mazhang 2013-12-26 06:16:46 UTC 
Sorry for reply so later. Just have a try, the bare-metal system can't boot from this nic, this device also not exist in "Boot sequence" of host BIOS, so not a bug.
Comment 8 Alex Williamson 2014-02-04 16:09:36 UTC 
Closing per comment #7, host also doesn't boot from NIC, so no evidence that we should be able to boot from this NIC.