Bug 1273196 - Q35 assigned devices report Code 10 on Win7 & Win8.1 guests in default config
Q35 assigned devices report Code 10 on Win7 & Win8.1 guests in default config
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
Unspecified Unspecified
high Severity unspecified
: rc
: ---
Assigned To: Alex Williamson
jingzhao
:
Depends On:
Blocks: 1305606 1313485 1227278
  Show dependency treegraph
 
Reported: 2015-10-19 18:04 EDT by Alex Williamson
Modified: 2016-11-07 15:47 EST (History)
11 users (show)

See Also:
Fixed In Version: qemu-kvm-rhev-2.5.0-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-07 15:47:27 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
the screen shot for the test result (303.04 KB, image/png)
2016-05-24 02:02 EDT, jingzhao
no flags Details
xml for VM that reproduces this issue (4.77 KB, text/plain)
2016-09-15 15:21 EDT, Alex Williamson
no flags Details

  None (edit)
Description Alex Williamson 2015-10-19 18:04:59 EDT
Description of problem:
Assigning an 82576 PF to a VM using virt-manager places the device on bus pci.2.  In this configuration both Windows 7 and Windows 8.1 report Code 10 (device cannot start).  RHEL7 guests work normally in this configuration.

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-29.el7.x86_64
kernel-3.10.0-322.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Attach an assigned device to a Windows 7 or 8.1 guest at the default location
2.
3.

Actual results:
Code 10

Expected results:
Device works

Additional info:
Comment 1 Alex Williamson 2015-10-20 14:18:31 EDT
Further testing with Quadro K4000 GPU and Intel 82576 PF NIC assigned to VMs.  Using q35 I placed both devices on the default bridge (pci.2, bus='0x02'), Win7 reported Code 12 for the GPU and Code 10 for the NIC.  Win8.1 reported Code 10 for both.  RHEL6.7 and RHEL7.2 guests work well with both devices.

Moved devices to pci.1, bus='0x01'.  Win7 and Win8.1 both reported Code 10 for all devices, RHEL6.7 and RHEL7.2 work well.

Moved devices to pcie.0, bus='0x00'.  All devices work with all guests.

Testing with a 440FX, Windows 7 VM, I assigned 82576 PF, K4000, and Teradici functions using pci.1, bus='0x01'.  All devices except the GPU work well, with the GPU reporting Code 12.

I think we should handle the Code 10s here, which seem to be unique to Q35 machine types, and the Code 12 in bug 1273172, which seems to require a Windows 7 guest with NVIDIA GPU behind a bridge.
Comment 2 Alex Williamson 2015-10-21 16:23:05 EDT
I took a wild guess and removed the PCI Express capability from the capability chain on the device, it seems to work.  Typically we leave this capability in place and things seem to work, at least on 440fx.  The behavior here seems to indicate that Windows behaves differently depending on whether the host bus is express (q35) or conventional (440fx).  I can work on some logic to test whether the root bus above a device is PCI-e and hide the express capability if the local bus is conventional, but it would be nice if we could contact Microsoft to understand their requirements.
Comment 3 Alex Williamson 2015-10-22 21:03:28 EDT
I think this resolves it:

http://lists.nongnu.org/archive/html/qemu-devel/2015-10/msg05385.html

tg3 is a casualty in this, there may be some devices that can only be supported on a PCIe configuration in the VM
Comment 4 Alex Williamson 2016-01-18 19:42:56 EST
This is already in QEMU2.5:

commit 0282abf078c3353a178ab77a115828ce333181dd
Author: Alex Williamson <alex.williamson@redhat.com>
Date:   Tue Nov 10 12:11:08 2015 -0700

    vfio/pci: Hide device PCIe capability on non-express buses for PCIe VMs
    
    When we have a PCIe VM, such as Q35, guests start to care more about
    valid configurations of devices relative to the VM view of the PCI
    topology.  Windows will error with a Code 10 for an assigned device if
    a PCIe capability is found for a device on a conventional bus.  We
    also have the possibility of IOMMUs, like VT-d, where the where the
    guest may be acutely aware of valid express capabilities on physical
    hardware.
    
    Some devices, like tg3 are adversely affected by this due to driver
    dependencies on the PCIe capability.  The only solution for such
    devices is to attach them to an express capable bus in the VM.
    
    Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Comment 6 jingzhao 2016-05-24 01:53:16 EDT
Hi Alex

  Didn't reproduce with kernel-3.10.0-322.el7.x86_64,qemu-kvm-rhev-2.3.0-29.el7.x86_64. Following is the reproduced steps:

1. Boot vm with following cli:
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 2,sockets=2,cores=1,threads=1 \
-enable-kvm \
-name rhel7 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-serial unix:/tmp/serial1,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-vga std \
-vnc :2 \
-qmp tcp:0:6667,server,nowait \
-chardev file,path=/home/seabios1.log,id=seabios1 \
-device isa-debugcon,chardev=seabios1,iobase=0x402 \
-monitor stdio \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=pcie.1 \
-device ioh3420,bus=pcie.0,addr=1c.1,port=2,chassis=2,id=pcie.2 \
-device virtio-scsi-pci,id=scsi,bus=pcie.1 \
-drive file=/home/win8.raw,if=none,id=drive-virtio-scsi-disk,format=raw,cache=none \
-device scsi-disk,bus=scsi.0,drive=drive-virtio-scsi-disk,id=virtio-scsi-disk,bootindex=0 \
-cdrom /usr/share/virtio-win/virtio-win-1.8.0.iso \
-device vfio-pci,host=03:00.0,bus=pcie.2,id=assin_nic \

2.check the 82576 nic in guest and didn't hit the Code10 error.

Could you help me to check the comdline, right?
Comment 7 jingzhao 2016-05-24 02:02 EDT
Created attachment 1160876 [details]
the screen shot for the test result
Comment 8 jingzhao 2016-05-31 23:26:23 EDT
Please check comment 6 and any comment for the issue?
Comment 9 Alex Williamson 2016-06-02 20:23:35 EDT
As noted in comment 0, the issue occurs when the device is attached to pci.2, which is behind the pci-bridge, behind the dmi-to-pci-bridge.  You are clearly placing the assigned device behind a pcie-root device.  I'm able to easily reproduce with the following xml:

<domain type='kvm'>
  <name>win8-q35-bz1273196</name>
  <uuid>9d06c9df-0a8b-40d9-be09-4b7ae9c7c431</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.2.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>IvyBridge</model>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/rhel/win8-q35'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:c1:84:7e'/>
      <source dev='enp1s0' mode='bridge'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Reproduced with qemu-kvm-rhev-2.3.0-31.el7_2.14.x86_64, verified with qemu-kvm-rhev-2.6.0-4.el7.x86_64, VM installed with en_windows_8.1_n_with_update_x64_dvd_4065114.iso
Comment 10 jingzhao 2016-06-03 05:47:22 EDT
Hi Alex

  Also didn't reproduce the issue, 82576 nic didn't be found in the guest, could you help to check it?  

[root@hp-z800-01 ~]# uname -r
3.10.0-322.el7.x86_64
[root@hp-z800-01 ~]# rpm -qa |grep qemu
ipxe-roms-qemu-20130517-7.gitc4bce43.el7.noarch
qemu-kvm-common-rhev-2.3.0-31.el7_2.14.x86_64
qemu-kvm-tools-rhev-2.3.0-31.el7_2.14.x86_64
qemu-img-rhev-2.3.0-31.el7_2.14.x86_64
libvirt-daemon-driver-qemu-1.3.4-1.el7.x86_64
qemu-kvm-rhev-debuginfo-2.3.0-31.el7_2.14.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.14.x86_64
[root@hp-z800-01 ~]# rpm -qa |grep seabios
seabios-1.7.5-10.el7.x86_64
seabios-bin-1.7.5-10.el7.noarch


[root@hp-z800-01 ~]# lspci -vvv -s 03:00.0
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
	Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
	Physical Slot: 1
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 29
	Region 0: Memory at e4800000 (32-bit, non-prefetchable) [disabled] [size=128K]
	Region 1: Memory at e4000000 (32-bit, non-prefetchable) [disabled] [size=4M]
	Region 2: I/O ports at c000 [disabled] [size=32]
	Region 3: Memory at e4840000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D3 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable- Count=10 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00002000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 16ms to 55ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [140 v1] Device Serial Number 00-1b-21-ff-ff-42-33-84
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 128, stride: 2, Device ID: 10ca
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000e4848000 (64-bit, non-prefetchable)
		Region 3: Memory at 00000000e4868000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Kernel driver in use: vfio-pci

1. First, used virsh start libvirt.xml. nic didn't be found in the win8.1 guest

[root@hp-z800-01 ~]# cat /home/libvirt.xml 
<domain type='kvm' id='3'>
  <name>win8-q35-bz1273196</name>
  <uuid>0703e9c4-0bd2-4486-8c33-b53b09bd3a59</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.2.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/home/win8.raw'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:c1:84:7e'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </hostdev>
    <memballoon model='none'>
      <alias name='balloon0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c785,c874</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c785,c874</imagelabel>
  </seclabel>
</domain>

2.Used the qemu command 
2.1) boot up the guest
[root@hp-z800-01 ~]# cat /home/bug1
/usr/libexec/qemu-kvm \
-name win8-q35-bz1273196,debug-threads=on \
-S \
-machine pc-q35-rhel7.2.0,accel=kvm,usb=off,vmport=off \
-cpu qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-m 8192 -realtime mlock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 0703e9c4-0bd2-4486-8c33-b53b09bd3a59 \
-no-user-config \
-nodefaults \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-no-shutdown \
-boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1.0x1 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-drive file=/home/win8.raw,format=raw,if=none,id=drive-sata0-0-2,cache=none,aio=native \
-device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=2 \
-drive if=none,media=cdrom,id=drive-sata0-0-1,readonly=on \
-device ide-cd,bus=ide.1,drive=drive-sata0-0-1,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=26,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:c1:84:7e,bus=pci.2,addr=0x1 \
-device usb-tablet,id=input0 \
-vnc 127.0.0.1:0 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.2,addr=0x2 \
-msg timestamp=on \
-monitor stdio \

2.2) check the nic with “info pci” but didn't found the nic

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:29c0
      id ""
  Bus  0, device   1, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR2: 32 bit memory at 0xfea10000 [0xfea10fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "video0"
  Bus  0, device  29, function 0:
    USB controller: PCI device 8086:2934
      IRQ 16.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device  29, function 1:
    USB controller: PCI device 8086:2935
      IRQ 17.
      BAR4: I/O at 0xc060 [0xc07f].
      id ""
  Bus  0, device  29, function 2:
    USB controller: PCI device 8086:2936
      IRQ 18.
      BAR4: I/O at 0xc080 [0xc09f].
      id ""
  Bus  0, device  29, function 7:
    USB controller: PCI device 8086:293a
      IRQ 19.
      BAR0: 32 bit memory at 0xfea11000 [0xfea11fff].
      id "usb"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfff00000, 0x000fffff]
      id "pci.1"
  Bus  1, device   1, function 1:
    PCI bridge: PCI device 1b36:0001
      IRQ 0.
      BUS 0.
      secondary bus 0.
      subordinate bus 0.
      IO range [0x0000, 0x0fff]
      memory range [0x00000000, 0x000fffff]
      prefetchable memory range [0x00000000, 0x000fffff]
      BAR0: 64 bit memory at 0xffffffffffffffff [0x000000fe].
      id "pci.2"
  Bus  0, device  31, function 0:
    ISA bridge: PCI device 8086:2918
      id ""
  Bus  0, device  31, function 2:
    SATA controller: PCI device 8086:2922
      IRQ 16.
      BAR4: I/O at 0xc0a0 [0xc0bf].
      BAR5: 32 bit memory at 0xfea12000 [0xfea12fff].
      id ""
  Bus  0, device  31, function 3:
    SMBus: PCI device 8086:2930
      IRQ 10.
      BAR4: I/O at 0x0700 [0x073f].
      id ""

2.3) check the nic through "info qtree" and nic can be found

dev: i82801b11-bridge, id "pci.1"
        addr = 1e.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        class PCI bridge, addr 00:1e.0, pci id 8086:244e (sub 0000:0000)
        bus: pci.1
          type PCI
          dev: pci-bridge, id "pci.2"
            chassis_nr = 2 (0x2)
            msi = true
            shpc = true
            addr = 01.1
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            class PCI bridge, addr 01:01.1, pci id 1b36:0001 (sub 0000:0000)
            bar 0: mem at 0xffffffffffffffff [0xfe]
            bus: pci.2
              type PCI
              dev: vfio-pci, id "hostdev0"
                host = "0000:03:00.0"
                x-intx-mmap-timeout-ms = 1100 (0x44c)
                x-vga = false
                x-req = true
                x-mmap = true
                addr = 02.0
                romfile = ""
                rombar = 1 (0x1)
                multifunction = false
                command_serr_enable = true
                class Ethernet controller, addr 00:02.0, pci id 8086:10c9 (sub 8086:a03c)
                bar 0: mem at 0xffffffffffffffff [0x1fffe]
                bar 1: mem at 0xffffffffffffffff [0x3ffffe]
                bar 2: i/o at 0xffffffffffffffff [0x1e]
                bar 3: mem at 0xffffffffffffffff [0x3ffe]



3. 82576 nic can be found when connect to pcie directly
 3.1) boot vm with following command:
[root@hp-z800-01 ~]# cat /home/bug2
/usr/libexec/qemu-kvm \
-name win8-q35-bz1273196,debug-threads=on \
-S \
-machine pc-q35-rhel7.2.0,accel=kvm,usb=off,vmport=off \
-cpu qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-m 8192 -realtime mlock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 0703e9c4-0bd2-4486-8c33-b53b09bd3a59 \
-no-user-config \
-nodefaults \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-no-shutdown \
-boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1.0x1 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-drive file=/home/win8.raw,format=raw,if=none,id=drive-sata0-0-2,cache=none,aio=native \
-device ide-hd,bus=ide.2,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=2 \
-drive if=none,media=cdrom,id=drive-sata0-0-1,readonly=on \
-device ide-cd,bus=ide.1,drive=drive-sata0-0-1,id=sata0-0-1,bootindex=1 \
-netdev tap,fd=26,id=hostnet0 \
-device e1000,netdev=hostnet0,id=net0,mac=52:54:00:c1:84:7e,bus=pci.2,addr=0x1 \
-device usb-tablet,id=input0 \
-vnc 127.0.0.1:0 \
-device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-device vfio-pci,host=03:00.0,id=hostdev0,bus=pcie.0,addr=0x2 \
-msg timestamp=on \
-monitor stdio \

3.2) check nic device through "info pci"

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:29c0
      id ""
  Bus  0, device   1, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfd000000 [0xfdffffff].
      BAR2: 32 bit memory at 0xfea10000 [0xfea10fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "video0"
  Bus  0, device   2, function 0:
    Ethernet controller: PCI device 8086:10c9
      IRQ 11.
      BAR0: 32 bit memory at 0xfe800000 [0xfe81ffff].
      BAR1: 32 bit memory at 0xfe400000 [0xfe7fffff].
      BAR2: I/O at 0xc040 [0xc05f].
      BAR3: 32 bit memory at 0xfe830000 [0xfe833fff].
      id "hostdev0"
  Bus  0, device  29, function 0:
    USB controller: PCI device 8086:2934
      IRQ 16.
      BAR4: I/O at 0xc040 [0xc05f].
      id ""
  Bus  0, device  29, function 1:
    USB controller: PCI device 8086:2935
      IRQ 17.
      BAR4: I/O at 0xc060 [0xc07f].
      id ""
  Bus  0, device  29, function 2:
    USB controller: PCI device 8086:2936
      IRQ 18.
      BAR4: I/O at 0xc080 [0xc09f].
      id ""
  Bus  0, device  29, function 7:
    USB controller: PCI device 8086:293a
      IRQ 19.
      BAR0: 32 bit memory at 0xfea11000 [0xfea11fff].
      id "usb"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 0.
      secondary bus 1.
      subordinate bus 1.
      IO range [0xf000, 0x0fff]
      memory range [0xfff00000, 0x000fffff]
      prefetchable memory range [0xfff00000, 0x000fffff]
      id "pci.1"
  Bus  1, device   1, function 1:
    PCI bridge: PCI device 1b36:0001
      IRQ 0.
      BUS 0.
      secondary bus 0.
      subordinate bus 0.
      IO range [0x0000, 0x0fff]
      memory range [0x00000000, 0x000fffff]
      prefetchable memory range [0x00000000, 0x000fffff]
      BAR0: 64 bit memory at 0xffffffffffffffff [0x000000fe].
      id "pci.2"
  Bus  0, device  31, function 0:
    ISA bridge: PCI device 8086:2918
      id ""
  Bus  0, device  31, function 2:
    SATA controller: PCI device 8086:2922
      IRQ 16.
      BAR4: I/O at 0xc0a0 [0xc0bf].
      BAR5: 32 bit memory at 0xfea12000 [0xfea12fff].
      id ""
  Bus  0, device  31, function 3:
    SMBus: PCI device 8086:2930
      IRQ 10.
      BAR4: I/O at 0x0700 [0x073f].
      id ""

My test environment and passwd:redhat
[root@hp-z800-01 home]# hostname
hp-z800-01.qe.lab.eng.nay.redhat.com
Comment 11 Alex Williamson 2016-06-03 11:25:42 EDT
(In reply to jingzhao from comment #10)
> My test environment and passwd:redhat
> [root@hp-z800-01 home]# hostname
> hp-z800-01.qe.lab.eng.nay.redhat.com

The VM named "win8.1" reproduces the issue on this system.  I don't know what is wrong with the VM named "win8-q35-bz1273196" on the system, but the guest not being able to see any devices downstream of the pci-to-dmi-bridge is beyond this scope of this bz.
Comment 12 jingzhao 2016-08-23 03:25:46 EDT
Hi Alex

  Can we marked as verified through Comment 9, because I didn't reproduce the bz on my side.

Thanks
Jing Zhao
Comment 13 Alex Williamson 2016-08-23 07:30:48 EDT
I provided a VM on your system that does reproduce the issue, as indicated in comment 11.  It is reproducible on the QE system and should therefore be reproduced and verified.
Comment 14 jingzhao 2016-09-14 06:36:01 EDT
Alex, could you provide the win8.1 iso which you used, because I didn't reproduce the bz on my side.

Thanks
Comment 15 Alex Williamson 2016-09-15 15:19:49 EDT
I don't understand why this is so difficult to reproduce, it works every time for me.  I used virt-manager to create a VM, my starting point is qemu-kvm-rhev-2.6.0-25 running on kernel 505.  I select to install from ISO image, for which I use en_windows_8.1_n_with_update_x64_dvd_4065114.iso from MSDN.  The guest OS selected for installation type is Windows 8.1.  Prior to installation I select the option to configure the VM.  I change the machine type to Q35, change the IDE and CDROM to SATA, and add an assigned 82576 PF NIC, entirely using the virt-manager interface.  I install the VM, selecting to disable updates.  When complete I have two NICs available, the emulated rtl8139 and the 82576, both are working and report normal in device manager.

At this point I shut down the VM, uninstall qemu-kvm-rhev 2.6.0:

rpm -e --nodeps qemu-kvm-common-rhev-2.6.0-25.el7.x86_64 qemu-img-rhev-2.6.0-25.el7.x86_64 qemu-kvm-rhev-2.6.0-25.el7.x86_64

Install qemu-kvm-rhev 2.3.0:

rpm -ihv qemu-img-rhev-2.3.0-31.el7_2.22.x86_64.rpm qemu-kvm-common-rhev-2.3.0-31.el7_2.22.x86_64.rpm qemu-kvm-rhev-2.3.0-31.el7_2.22.x86_64.rpm

Then I edit the XML only to change the machine type from:

"pc-q35-rhel7.3.0"

to

"pc-q35-rhel7.2.0"

I boot the VM again and now I only have one working NIC available, the rtl8139.  Device manager shows a yellow exclamation next to the 82576 PF NIC and reports Code 10 in the properties dialog.

Note, I did not install anything extra beyond the Windows installation media noted above.  I'll attach the XML for my VM.
Comment 16 Alex Williamson 2016-09-15 15:21 EDT
Created attachment 1201371 [details]
xml for VM that reproduces this issue
Comment 17 jingzhao 2016-09-17 22:11:37 EDT
Reproduce it on qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64. and Verified it on qemu-kvm-rhev-2.6.0-25.el7.x86_64. 

The difference between Alex's steps and mine is add 82576 nic when install the win8.1 system. 

/usr/libexec/qemu-kvm \
-name win8-q35-bz1273196,debug-threads=on \
-machine q35,accel=kvm,usb=off,vmport=off \
-cpu qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-m 4G -realtime mlock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-uuid 0703e9c4-0bd2-4486-8c33-b53b09bd3a59 \
-no-user-config \
-nodefaults \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-usb \
-no-shutdown \
-boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0 \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1 \
-device ahci,id=ahci0 \
-drive file=/home/bug/win8.1.raw,format=raw,if=none,id=drive-sata0-0-2,cache=none,aio=native \
-device ide-hd,bus=ahci0.0,drive=drive-sata0-0-2,id=sata0-0-2,bootindex=2 \
-drive if=none,media=cdrom,id=drive-sata0-0-1,readonly=on,file=/home/bug/en_windows_8_1_enterprise_x64_dvd_2971902.iso \
-device ide-cd,bus=ahci0.1,drive=drive-sata0-0-1,id=sata0-0-1,bootindex=1 \
-vga qxl \
-vnc :0 \
-device vfio-pci,host=03:00.0,id=vf-00.0,bus=pci.2,addr=0x1.0 \
-netdev tap,id=hostnet1 \
-device rtl8139,netdev=hostnet1,id=net1,mac=54:52:00:B6:40:22 \
-cdrom /home/bug/virtio-win.iso \
-msg timestamp=on \
-monitor stdio \

Thanks
Jing Zhao
Comment 19 errata-xmlrpc 2016-11-07 15:47:27 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html

Note You need to log in before you can comment on or make changes to this bug.