Bug 1434321

Hi,
Can you please try to reproduce with:

1. windows 2016 on q35 & Seabios
2. fedora on q35 & ovmf


Thanks,
Marcel

Please see comment #2

Thanks Marcel for the CC.

I agree with the additional info requests in comment 2.

Also, please upload
- the "lspci -v -v -v" output from the host side, for the device being assigned, before attempting the assignment,
- and the OVMF debug log.

These two items should be standard parts of any OVMF / device assignment bug report. Thanks!

OVMF log, please check the attachment
82599ES nic info, pleas check following

[root@dell-per730-29 win]# lspci -vvv -s 07:00.1
07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Subsystem: Intel Corporation Ethernet Server Adapter X520-2
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin B routed to IRQ 56
	NUMA node: 0
	Region 0: Memory at 94c00000 (64-bit, non-prefetchable) [size=512K]
	Region 2: I/O ports at 2000 [size=32]
	Region 4: Memory at 94d00000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
		Not readable
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-c3-d0-3c
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable+ Migration- Interrupt- MSE+ ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 8, Function Dependency Link: 01
		VF offset: 128, stride: 2, Device ID: 10ed
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000094d04000 (64-bit, non-prefetchable)
		Region 3: Memory at 0000000094e04000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: ixgbe
	Kernel modules: ixgbe

Thanks
Jing

Created attachment 1265563 [details]
ovmf log

Hi Marcel

  1. Reproduce the issue on windown2016 with q35 & seabios.
  2. Didn't reproduce the issue on fedora 24 with q35 & ovmf

Thanks

Created attachment 1265568 [details]
2016_q35_seabios log

Created attachment 1265569 [details]
fedora_ovmf_log_q35

Thanks a lot for the test results; so we have:

                           Q35+Seabios       Q35+OVMF
Windows 2016 Server guest  fail (comment 7)  fail (comment 0)
Fedora 24 guest            ?                 pass (comment 7)

The distinguishing factor seems to be Windows vs. Fedora; the firmware appears to be neutral. Is this an ACPI issue maybe? Or a problem with the Windows VF driver?

--*--

Furthermore, I tried to correlate the OVMF log with the data available in the report. Unfortunately, I can't do that:

- In the OVMF log, I don't see the actual device being assigned -- I can see no MMIO BAR with size 512KB == 0x80000! (See this BAR size in comment 5.)

- In the OVMF log, I can also see no device on the bridge that has address 0xa.2 (see this address, for "root2", in comment 0), i.e., on the bridge that is supposed to carry the assigned device. In the OVMF log, we have three non-root bridges, at [00|0A|00], [00|0A|01] and [00|0A|02], providing secondary buses 2, 3 and <none>, and only the first two bridges have devices on them (one on each), [00|0A|02] is empty.

To me it looks like the OVMF log in comment 5 is inconsistent with the report in comment 0. However it may be though, I see no allocation errors in the OVMF log.

Hi,

Do you know if is a regression? (Is there any version of QEMU/Windows that actually works?)

Thanks,
Marcel

(In reply to Marcel Apfelbaum from comment #11)
> Hi,
> 
> Do you know if is a regression? (Is there any version of QEMU/Windows that
> actually works?)
> 
> Thanks,
> Marcel

Hi Marcel

I think it's not a regression. the bz also can be reproduced with qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64. 

Didn't reproduce the bz on pc machine type.


Thanks
Jing

Hi all,

I am currently trying to reproduce the issue but I am facing an issue. I have executed the VF commands as written in "steps to reproduce", but I keep getting the following error:

qemu-system-x86_64: -device vfio-pci,host=0000:02:10.7,id=vf1,bus=root2: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are  bound to their vfio bus driver.

As far as I understand we should attach all the devices on the same iommu_group to the vfio driver in order to pass a certain device to Qemu. The problem is that all of the devices (VF and physical) share the same iommu_group id of 1, if we attempt to unbind the ixgbe intel driver from the physical device, all of the Virtual Functions will disappear and we are stuck in an endless loop! How this should be performed without getting the error above?

> The problem is that all of the devices (VF and physical) share the same
> iommu_group id of 1

Your host system is unsuitable for working with device assignment, either
entirely, or in its current (physical) configuration. Please refer to Q1 in
<http://vfio.blogspot.hu/2014/08/vfiovga-faq.html>:

> Question 1:
>
> I get the following error when attempting to start the guest:
>
>   vfio: error, group $GROUP is not viable, please ensure all devices
>   within the iommu_group are bound to their vfio bus driver.
>
> Answer:
>
> There are more devices in the IOMMU group than you're assigning, they all
> need to be bound to the vfio bus driver (vfio-pci) or pci-stub for the
> group to be viable.  See my previous post about IOMMU groups for more
> information.  To reduce the size of the IOMMU group, install the device
> into a different slot, try a platform that has better isolation support,
> or (at your own risk) bypass ACS using the ACS override patch.

The "previous post" referenced above is
<http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html>.

Can we get some clarification on this bug, we have:

(In reply to jingzhao from comment #0)
> Created attachment 1264955 [details]
> screenshot of win2016
> 
> Description of problem:
> Code 10 error when install VF driver in windows 2016 on q35 & ovmf

"VF driver"...  The Windows Device Manager dialog also clearly shows "Virtual Function" as part of the device name.

> Steps to Reproduce:
> Pre: create vf in host
> echo 8 >/sys/bus/pci/devices/0000\:07\:00.1/sriov_numvfs
> echo 0000:07:10.1 >/sys/bus/pci/devices/0000\:07\:10.1/driver/unbind 
> echo "8086 10ed" >/sys/bus/pci/drivers/vfio-pci/new_id 
> echo "8086 10ed" >/sys/bus/pci/drivers/vfio-pci/remove_id 

This created VF, apparently including 07:10.1, which is then bound to vfio-pci...

> [root@dell-per730-29 win]# ethtool -i p5p2
> driver: ixgbe
> version: 4.4.0-k-rh7.4
> firmware-version: 0x546c0001
> expansion-rom-version: 
> bus-info: 0000:07:00.1
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no

So we're being provided with the PF NIC info.

> [2]qemu command line:
> /usr/libexec/qemu-kvm \
...
> -device ioh3420,bus=pcie.0,id=root2,chassis=12,addr=0xa.2 \
> -device ioh3420,bus=pcie.0,id=root3 \
> -device vfio-pci,host=0000:07:00.1,id=vf1,bus=root2 \

And this indicates the _PF_ is being assigned.

(In reply to jingzhao from comment #5)
> OVMF log, please check the attachment
> 82599ES nic info, pleas check following
> 
> [root@dell-per730-29 win]# lspci -vvv -s 07:00.1
> 07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
> Network Connection (rev 01)

Here we have the _PF_ lspci info in response to a request for the lspci info for_the_device_being_assigned.

(In reply to Laszlo Ersek from comment #10)
> - In the OVMF log, I don't see the actual device being assigned -- I can see
> no MMIO BAR with size 512KB == 0x80000! (See this BAR size in comment 5.)
... 
> To me it looks like the OVMF log in comment 5 is inconsistent with the
> report in comment 0. However it may be though, I see no allocation errors in
> the OVMF log.

It is, because comment 5 is for the PF, 82599 VF have two 16K BARs as the SeaBIOS log shows:

PCI: map device bdf=01:00.0  bar 0, addr fc000000, size 00004000 [mem]
PCI: map device bdf=01:00.0  bar 3, addr fc004000, size 00004000 [mem]
...
PCI: init bdf=01:00.0 id=8086:10ed

Note that 0x10ed is a _VF_ device ID.

So comment 5 is mostly irrelevant and the QEMU command line is given incorrectly.  An ixgbe VF looks more like this:

01:10.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)
	Subsystem: Dell Device 1f61
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Region 0: [virtual] Memory at d8c00000 (64-bit, non-prefetchable) [size=16K]
	Region 3: [virtual] Memory at d8d00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [70] MSI-X: Enable- Count=3 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [1d0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

N.B. the lspci info in comment 15 is for an X540, which seems to have added ACS on both the PF and VF, the capability is missing on 82599, where isolation is handled via an ACS equivalent quirk.  We do mask ARI for assigned devices, which means the guest should see standard capabilities including MSI-X and PCIe, with AER being the lone visible extended capability.  Sameeh's results seem to confirm this is the case.

See comment 15, is the VF or PF being assigned?  If VF, where is the VF MAC address being set?  I'm unable to reproduce on an X540 VF managed by libvirt (q35+OVMF):

kernel-3.10.0-663.el7.x86_64
qemu-kvm-rhev-2.9.0-3.el7.x86_64
OVMF-20170228-4.gitc325e41585e3.el7.noarch

Is this still reproduced with QEMU 2.9 on 82599?  Both the built-in Windows driver and Intel ProSet 22.3 works on my X540 VF.

(In reply to Alex Williamson from comment #17)
> See comment 15, is the VF or PF being assigned?  If VF, where is the VF MAC
> address being set?  I'm unable to reproduce on an X540 VF managed by libvirt
> (q35+OVMF):
> 
> kernel-3.10.0-663.el7.x86_64
> qemu-kvm-rhev-2.9.0-3.el7.x86_64
> OVMF-20170228-4.gitc325e41585e3.el7.noarch
> 
> Is this still reproduced with QEMU 2.9 on 82599?  Both the built-in Windows
> driver and Intel ProSet 22.3 works on my X540 VF.

Hi Alex

Also reproduce the bz on my side.

Host: kernel-3.10.0-664.el7.x86_64
qemu-kvm-rhev-2.9.0-3.el7.x86_64
OVMF-20170228-4.gitc325e41585e3.el7.noarch

Win2016 iso:en_windows_server_2016_x64_dvd_9718492.iso

Used 82599 nic on my side.

Thanks
Jing

(In reply to jingzhao from comment #18)
> (In reply to Alex Williamson from comment #17)
> > See comment 15, is the VF or PF being assigned?
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> > If VF, where is the VF MAC address being set?
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

> Also reproduce the bz on my side.
> 
> Host: kernel-3.10.0-664.el7.x86_64
> qemu-kvm-rhev-2.9.0-3.el7.x86_64
> OVMF-20170228-4.gitc325e41585e3.el7.noarch
> 
> Win2016 iso:en_windows_server_2016_x64_dvd_9718492.iso
> 
> Used 82599 nic on my side.

Does Windows have a built-in driver for the device?  Does it work?

(In reply to Alex Williamson from comment #19)
> (In reply to jingzhao from comment #18)
> > (In reply to Alex Williamson from comment #17)
> > > See comment 15, is the VF or PF being assigned?
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> > > If VF, where is the VF MAC address being set?
>     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> > Also reproduce the bz on my side.
> > 
> > Host: kernel-3.10.0-664.el7.x86_64
> > qemu-kvm-rhev-2.9.0-3.el7.x86_64
> > OVMF-20170228-4.gitc325e41585e3.el7.noarch
> > 
> > Win2016 iso:en_windows_server_2016_x64_dvd_9718492.iso
> > 
> > Used 82599 nic on my side.
> 
> Does Windows have a built-in driver for the device?  Does it work?

No built-driver, so I used the driver "https://downloadcenter.intel.com/download/26092/Intel-Network-Adapter-Driver-for-Windows-Server-2016-?product=83418"

Thanks
Jing

(In reply to jingzhao from comment #20)
> (In reply to Alex Williamson from comment #19)
> > (In reply to jingzhao from comment #18)
> > > (In reply to Alex Williamson from comment #17)
> > > > See comment 15, is the VF or PF being assigned?
> >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > 
> > > > If VF, where is the VF MAC address being set?
> >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Please answer these questions.

(In reply to Alex Williamson from comment #21)
> (In reply to jingzhao from comment #20)
> > (In reply to Alex Williamson from comment #19)
> > > (In reply to jingzhao from comment #18)
> > > > (In reply to Alex Williamson from comment #17)
> > > > > See comment 15, is the VF or PF being assigned?
> > >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Used VF on my side

> > > 
> > > > > If VF, where is the VF MAC address being set?
> > >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[root@dell-per730-29 ~]# ip link show p5p1
5: p5p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
    link/ether 00:1b:21:c3:d0:3c brd ff:ff:ff:ff:ff:ff
    vf 0 MAC ee:e3:cf:60:be:cc, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 6a:ca:4d:eb:46:0a, spoof checking on, link-state auto, trust off, query_rss off
    vf 2 MAC 7a:e0:8e:a3:49:9a, spoof checking on, link-state auto, trust off, query_rss off
    vf 3 MAC 0a:4f:6e:5e:19:84, spoof checking on, link-state auto, trust off, query_rss off
    vf 4 MAC 62:92:81:a4:3e:80, spoof checking on, link-state auto, trust off, query_rss off
    vf 5 MAC ee:a0:da:ab:47:ec, spoof checking on, link-state auto, trust off, query_rss off


May 12 01:18:57 localhost kernel: ixgbe 0000:07:00.0 p5p1: VF Reset msg received from vf 0
May 12 01:18:57 localhost kernel: ixgbe 0000:07:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.0: MAC address not assigned by administrator.
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.0: Assigning random MAC address
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.0: ee:e3:cf:60:be:cc
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.0: MAC: 1
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.0: Intel(R) 82599 Virtual Function
May 12 01:18:57 localhost kernel: ixgbevf 0000:07:10.2: enabling device (0000 -> 0002)
May 12 01:18:57 localhost kernel: ixgbe 0000:07:00.0 p5p1: VF Reset msg received from vf 1

Details, please check the /var/log/message, see the attachment

> 
> Please answer these questions.

Created attachment 1278069 [details]
/var/log/message

qemu command line of comment 22 used

/usr/libexec/qemu-kvm \
-machine q35,smm=on,accel=kvm \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=/home/OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
-serial unix:/tmp/serial0,server,nowait \
-debugcon file:/home/ovmf.log \
-global isa-debugcon.iobase=0x402 \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-vga qxl \
-global driver=cfi.pflash01,property=secure,value=on \
-drive file=/home/ovmf-win2016-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=0 \
-device ioh3420,bus=pcie.0,id=root0,multifunction=on,chassis=1,addr=0xa.0 \
-device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e,bus=root0 -netdev tap,id=tap10 \
-device ioh3420,bus=pcie.0,id=root1,chassis=11,addr=0xa.1 \
-device ioh3420,bus=pcie.0,id=root2,chassis=12,addr=0xa.2 \
-device ioh3420,bus=pcie.0,id=root3 \
-device vfio-pci,host=0000:07:10.0,id=vf1,bus=root2 \
-device ahci,id=ahci0 \
-drive file=/usr/share/virtio-win/virtio-win-1.9.0.iso,if=none,id=drive-virtio-disk1,format=raw \
-device ide-cd,drive=drive-virtio-disk1,id=virtio-disk1,bus=ahci0.0 \
-monitor stdio \
-vnc :1 \

Thanks
Jing

The root cause is that 82599 VF has invalid PCIE capability structure, where the version field set to 0. Windows on device enumeration (usually on startup) traverses the chain of capabilities and (if reads zero from version field) fails PNP start process. This does not depend on the driver of this specific device but rather on PCI stack as I can see.

Simple workaround is to remove in QEMU this invalid capability structure from the chain - we tried such patch for QEMU, then the device starts OK in both Windows or Linux guests. I think this will not be recognized as solution, as the negative side effect is that in this case Windows does not access configuration space above offset 100h, i.e. actually ignores all extended capabilities.

So, possible solution would be for this device (or such devices) use emulation of PCIE structure as implemented for fully emulated PCIE devices, as it is done in qemu/hw/pci/pcie.c, whose code can be reused.

We can prepare such change in QEMU, but prefer first to have a comment of Alex Williamson, as he may have different plan for this issue.

Created attachment 1292073 [details]
Emulate bits to bump express capability from v0 to v1

Does this patch resolve the issue?  vfio-pci already has support for emulating specific bits in config space, so we can detect this condition and fix it for all devices.  Re-assign to me if this works.

Yes, with this patch the device can start and we can see the Windows reads version field and then reads more words from the PCIE structure. Also accesses to config space 0x100 area present.
Reassigning to alex.williamson

Fixed in:

commit 47985727e383eee191a39cc007541c17cd8680ee
Author: Alex Williamson <alex.williamson>
Date:   Mon Jul 10 10:39:43 2017 -0600

    vfio/pci: Fixup v0 PCIe capabilities
    
    Intel 82599 VFs report a PCIe capability version of 0, which is
    invalid.  The earliest version of the PCIe spec used version 1.  This
    causes Windows to fail startup on the device and it will be disabled
    with error code 10.  Our choices are either to drop the PCIe cap on
    such devices, which has the side effect of likely preventing the guest
    from discovering any extended capabilities, or performing a fixup to
    update the capability to the earliest valid version.  This implements
    the latter.
    
    Signed-off-by: Alex Williamson <alex.williamson>


Included in:

$ git tag --contains 47985727e383eee191a39cc007541c17cd8680ee
v2.10.0
v2.10.0-rc0
v2.10.0-rc1
v2.10.0-rc2
v2.10.0-rc3
v2.10.0-rc4

Reproduce on qemu-kvm-rhev-2.9.0-16.el7_4.9.x86_64 and verified on qemu-kvm-rhev-2.10.0-3.el7.x86_64.

Other version:
OVMF-20171011-1.git92d07e48907f.el7.noarch
kernel-3.10.0-731.el7.x86_64

According to above test result,changed to verified status

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1104