You mention "ovmf_gpu.log" in "Additional info" in comment 0 -- can you please actually upload that file? :)

Also, Alex will certainly want to know the PCI vendorid/devid of the cards you are assigning. Thanks.

Created attachment 1198946 [details]
ovmf_gpu.log

GPU info:
1.#lspci (only GPU device)
   03:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
   03:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
   04:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
   04:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)

***just use 03:00.0 & 03:00.1 in test.****
2.# lspci -vvv -s 03:00.0
03:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: NVIDIA Corporation Device 0965
	Physical Slot: 6
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 37
	NUMA node: 0
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
	Region 3: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=32M]
	Region 5: I/O ports at 3000 [disabled] [size=128]
	Expansion ROM at f5080000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 1024 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] #19
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

3.# lspci -vvv -s 03:00.1
03:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
	Subsystem: NVIDIA Corporation Device 0965
	Physical Slot: 6
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin B routed to IRQ 38
	NUMA node: 0
	Region 0: Memory at f5000000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 1024 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest+
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

The Quadro GPU has a PCI expansion ROM that contains a UEFI driver (probably a GOP driver). Namely, in comment 5, you find

	Expansion ROM at f5080000 [disabled] [size=512K]

and in the OVMF log in comment 4, you see

> Loading driver at 0x0007E1DE000 EntryPoint=0x0007E1E01E8 

This lines follows the PCI enumeration / resource allocation closely, plus it does not provide a driver file name (a basename). All other such lines (that load drivers from the OVMF firmware binary) say "XXXX.efi" at the end of the line, so this is certainly the Quadro's oprom.

Now, from the end of the OVMF log in comment 4, it looks like the firmware hangs when it asks the GPU's option ROM to bind the device. Note that the same happens to QXL first, but that one works okay:

> Found PCI VGA device
> QemuVideo: QEMU QXL VGA detected
> [...]
> Found PCI VGA device
> [hang]

If this were to succeed, you would get the TianoCore splash screen not just on the QXL display, but also on your physical monitor. However, it doesn't succed.

So, obviously, my first guess is that the UEFI oprom on the Quadro is broken. Therefore, you please repeat your test with the following single modification:

  -device vfio-pci,host=03:00.0,id=GPU1,rombar=0
                                       ^^^^^^^^^

Thanks
Laszlo

Hi, Laszlo
  
   It really works well when using your cmds "-device vfio-pci,host=03:00.0,id=GPU1,rombar=0".


   Is it a hardware problem? Any solution?


Thanks,
aliang

(In reply to aihua liang from comment #7)
> Hi, Laszlo
>   
>    It really works well when using your cmds "-device
> vfio-pci,host=03:00.0,id=GPU1,rombar=0".

Thank you for confirming!

>    Is it a hardware problem?

It is certainly a problem with the option ROM that was flashed to the card by NVIDIA. So, I would call it a "firmware problem with the GPU".

> Any solution?

If NVIDIA provides a firmware upgrade for the card, we should try that. If they are willing to collaborate on debugging the problem in some other way, we should try that. Else, perhaps Alex can come up with a VFIO quirk in QEMU that automatically disables the oprom for this specific PCI vendorid/deviceid.

Got it, thanks laszlo.
I'll take the machine and try tomorrow, then i'll give a response.

I was hoping our GPU assignment documentation recommended rombar=0, but unfortunately it does not.  The GPU is not expected to work until the NVIDIA driver within the guest loads, so we have absolutely no requirement to support the ROM, but we also have no way to specifically detect supported cards vs unsupported card based only on PCI details like device ID without resorting to a blacklist nightmare.  It's possible that the UEFI ROM is making use of various backdoors into PCI configuration space of the device which is not virtualized.  I'll see if I can reproduce and trace it.  rombar=0 is a perfectly valid workaround to document for customers though.

Thank you very much Alex for the investigation. If your repro/tracing produces committable results for QEMU, I think that would be beneficial. Getting the docs in order about this looks more urgent though.

So, I'm thinking about reassigning this BZ to either the virt docs component, or maybe even libvirt (or the virt-tools such as virt-manager). After all, those components could implement a default of <rom bar='off'/> for all <hostdev> devices -- perhaps even for <interface type='hostdev'> (which is called the "intelligent" form of NIC assignment, because it allows one to override the MAC).

Thoughts? I'm CC'ing Laine and Andrea. Thanks!

Ah wait, you didn't speak about changing the *default* for rombar, you said it would be worth mentioning as a workaround in the docs. Sorry about the confusion. I'll reassign this to virt-docs then. Thanks!

(In reply to Laszlo Ersek from comment #13)
> Ah wait, you didn't speak about changing the *default* for rombar, you said
> it would be worth mentioning as a workaround in the docs. Sorry about the
> confusion. I'll reassign this to virt-docs then. Thanks!

Right, I am absolutely not suggesting any sort of global defaults change based on this, only documenting it for this particular use case.  We may be able to fix it somewhere else in the stack, but there's no purpose in passing the ROM on these particular devices for the use case supported in RHEL.

Moving BZ to Virtualization Deployment and Administration Guide:

In
- ⁠Chapter 17. "Guest Virtual Machine Device Configuration", 
- section 17.1.1. Assigning a PCI Device with virsh,
- step 4. "Add configuration details",

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/Virtualization_Deployment_and_Administration_Guide/chap-Guest_virtual_machine_device_configuration.html#sect-PCI_devices-Assigning_a_PCI_device_with_virsh

please append the following text:

  Assigned PCI devices, especially GPUs, may have expansion ROMs that do not
  cooperate with virtualization. This may result in boot failures for the guest
  operating system. The GPU is not expected to work until the native guest OS
  driver loads, therefore the recommended workaround is the following:

  - On the host, ascertain that the device to assign has an expansion ROM BAR.
    Run "lspci -v" for the device, and check the output for

    Expansion ROM at ...

  - Add the <rom bar='off'/> element as a child of the <hostdev> element:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
         <address domain='0' bus='0' slot='25' function='0'/>
      </source>
      <rom bar='off'/>
    </hostdev>

Thanks.

Hmm, on my K4000 this works correctly, I get the TianoCore splash on both the spice and physical monitor, net boot goes to spice.

(In reply to Laszlo Ersek from comment #15)

>   This may result in boot failures for the guest operating system.

s/guest operating system/guest firmware/

(In reply to Alex Williamson from comment #17)
> Hmm, on my K4000 this works correctly, I get the TianoCore splash on both
> the spice and physical monitor, net boot goes to spice.

I found random forum posts and webshop reviews that complained about K5000, saying that it only worked in specific slots on various motherboards. Perhaps the same is true for guests -- maybe the issue surfaces with the K5000 only if you place it in the "wrong" slots on Q35.

Created attachment 1199184 [details]
Log from working K4000, TianoCore boot splash on NVIDIA + SPICE

Right, it proceeds after the second "Found PCI VGA device" line.

And if I translate the well-known protocol GUIDs there, to protocol names, I get:

> Found PCI VGA device
> InstallProtocolInterface: FE47349A-7F0D-4641-822B-34BAA28ECDD0 3EA06698
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E968F30
> InstallProtocolInterface: CC0A2170-0B39-45D8-B469-D04E57B522B3 3E968C98
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E968A30
> InstallProtocolInterface: CC0A2170-0B39-45D8-B469-D04E57B522B3 3E968718
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E968630
> InstallProtocolInterface: CC0A2170-0B39-45D8-B469-D04E57B522B3 3E968318
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E967030
> InstallProtocolInterface: CC0A2170-0B39-45D8-B469-D04E57B522B3 3E967C18
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E967D30
> InstallProtocolInterface: CC0A2170-0B39-45D8-B469-D04E57B522B3 3E967918
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E967730
> InstallProtocolInterface: F5E527EE-A283-4127-9A2F-AFFABF26DF70 3E967418
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966F30
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966F30
> InstallProtocolInterface: [EfiEdidDiscoveredProtocol] 3E966C18
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966830
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966830
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966830
> InstallProtocolInterface: [EfiDevicePathProtocol] 3E966830
> InstallProtocolInterface: [EfiEdidActiveProtocol] 3E966A18
> InstallProtocolInterface: [EfiGraphicsOutputProtocol] 3E966318

Note the last of them: EfiGraphicsOutputProtocol (GUID 9042A9DE-23DC-4A38-96FB-7ADED080516A). So, this is both fine for the K4000, and is consistent with the analysis for the K5000. Thanks!

The rom on my K4000 is 80.06.53.00.01.  I also pulled 80.06.19.00.01 from techpowerup[1] and booted the VM with romfile= to use this rom, in this case the VM still boots and SPICE displays the bootsplash, but the monitor remains dark.

Aihua, could you try dumping the rom from your card to get the version info?  To do so (with the VM not running):

1) setpci -s 3:00.0 COMMAND=2:2
2) echo 1 > /sys/bus/pci/devices/0000:03:00.0/rom
3) cat /sys/bus/pci/devices/0000:03:00.0/rom > ~/K5000.rom
4) echo 0 > /sys/bus/pci/devices/0000:03:00.0/rom
5) setpci -s 3:00.0 COMMAND=0:2

This should give you a non-zero sized K5000.rom.  Run 'xxd K5000.rom | less' and in the right most column in the partial text area you should see "Version" followed by numbers similar to mine above.  Please report that version information.

Also, if not the same version, you may want to try the version on techpowerup[2].  They list the version there as 80.04.59.00.18.  I can't predict whether this will be newer/older/same as what you have, but a test using romfile= on the GPU function zero (only) may provide another data point.

[1] https://www.techpowerup.com/vgabios/?architecture=NVIDIA&manufacturer=NVIDIA&model=Quadro+K4000&interface=&memType=&memSize=

[2] https://www.techpowerup.com/vgabios/?architecture=NVIDIA&manufacturer=NVIDIA&model=Quadro+K5000&interface=&memType=&memSize=

nouveau is not blacklisted on the host, this is not the recommended configuration.  This can interfere with the operation of the GPU in the guest.  In fact, I don't know how you're even assigning this device as performing nodedev-detach on it results in soft lockups in the kernel and never acquires the device.  How are you doing device assignment on this GPU?  I'd suggest adding pci-stub.ids=10de:11ba to the host kernel configuration.  I see that the boot vga device is identical, so this will also interfere with host graphics if any are used.  If necessary to maintain host graphics, there are ways to exclude only the assigned graphics using the driver_override option and modprobe install scripts as discussed here: http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html  See paragraph beginning "Another issue that users encounter when sequestering devices is what to do when there are multiple devices with the same vendor:device ID..."

The current configuration of this host is invalid for assigning the GPU at 03:00.0.

Hi,Alex
 
 The rom on K5000 is: 80.04.59.00.18.

Execute Steps: 
 1.Assign K5000 to guest.
 2.In host,execute:
    1) setpci -s 3:00.0 COMMAND=2:2
    2) echo 1 > /sys/bus/pci/devices/0000:03:00.0/rom
    3) cat /sys/bus/pci/devices/0000:03:00.0/rom > ~/K5000.rom
    4) echo 0 > /sys/bus/pci/devices/0000:03:00.0/rom
    5) setpci -s 3:00.0 COMMAND=0:2
    6) xxd K5000.rom | less

Hi, Alex

   If i want to update K5000's rom version to "80.04.E2.00.15", should i update it on host? or on guest?
   Should i update it under vfio test environment?

Look forward for your reply.


Thanks,
aliang

See comment 28, your system is not properly configured for GPU assignment.  The nouveau driver should not be used in the host for the assigned GPU.  I logged into your system and tried a simple nodedev-detach which resulted in soft lockups trying to unbind the GPU from nouveau.  Fix these problems first.

Now,System is under GPU assignment(with only one GPU device plugged)

Details，as bellow:

 1. Enable intel_iommu

 2. modprobe vfio/vfio-pci

 3. # lspci -n -s 03:00.0
         03:00.0 0300: 10de:11ba (rev a1)
    # lspci -n -s 03:00.1
         03:00.1 0403: 10de:0e0a (rev a1)
    # echo 0000:03:00.0 >  /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind 
    # echo 0000:03:00.1 >  /sys/bus/pci/devices/0000\:03\:00.1/driver/unbind 
    # echo "10de 11ba" >  /sys/bus/pci/drivers/vfio-pci/new_id
    # echo "10de 11ba" >  /sys/bus/pci/drivers/vfio-pci/remove_id 
    # echo "10de 0e0a" >  /sys/bus/pci/drivers/vfio-pci/new_id
    # echo "10de 0e0a" >  /sys/bus/pci/drivers/vfio-pci/remove_id

  4. Then we can start guest with vfio-pci configured.(/home/ovmf_gpu.txt)
/usr/libexec/qemu-kvm -name rhel7 \
-machine q35,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 10240 \
-realtime mlock=off \
-smp 3 \
-uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
-no-user-config -nodefaults \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
-no-hpet \
-boot menu=on,splash-time=12000 \
-device vfio-pci,host=03:00.0,id=GPU1 \
-device vfio-pci,host=03:00.1,id=GPU2 \
-vga qxl \
-monitor stdio \
-vnc 0:2 \
-netdev tap,id=hostnet10 \
-device virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
-spice ipv4,port=5000,disable-ticketing \
-enable-kvm \
-global ICH9-LPC.disable_s3=0 \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
-drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
-device ahci,id=ahci \
-drive file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,format=qcow2,cache=none,aio=native \
-device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
-global driver=cfi.pflash01,property=secure,value=on \
-chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
-device isa-debugcon,chardev=ovmf_log,iobase=0x402 \


Result:
 guest start with dark screen & error info displayed in host:

"qemu-kvm: vfio_err_notifier_handler(0000:03:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
qemu-kvm: vfio_err_notifier_handler(0000:03:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest"

The system is actually in a worse configuration now, the assigned GPU is now the host boot graphics and nothing has been done to prevent nouveau from binding to the assigned GPU.  Add back the other K5000 or preferably a different model GPU.  Make sure the GPU for assignment is not the host boot graphics devices and prevent nouveau from binding to the assigned GPU.  You can do this with pci-stub.ids=10de:11ba on the host kernel commandline, but note that this will affect the other K5000, if installed, as well.  As noted in comment 28, there are ways to work around this for users where host graphics need to work as well.

Test Environment with K5000 was destroyed, so can't give more info now.

Additional Info:
1. When test with K2(no audio device), the problem not exist.
  qemu-cmds:
/usr/libexec/qemu-kvm -name rhel7 \
-machine q35 \
-cpu host \
-m 10240 \
-realtime mlock=off \
-smp 3 \
-uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
-no-user-config -nodefaults \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
-no-hpet \
-boot menu=on,splash-time=12000 \
-vga qxl \
-monitor stdio \
-vnc 0:2 \
-netdev tap,id=hostnet1 \
-device virtio-net-pci,netdev=hostnet1,id=net1,mac=58:54:00:49:b2:5f \
-spice ipv4,port=5000,disable-ticketing \
-enable-kvm \
-global ICH9-LPC.disable_s3=0 \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
-drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
-device ahci,id=ahci \
-drive file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,format=qcow2,cache=none,aio=native \
-device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
-global driver=cfi.pflash01,property=secure,value=on \
-chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
-device isa-debugcon,chardev=ovmf_log,iobase=0x402 \
-device vfio-pci,host=84:00.0,id=GPU1 \


2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio Controller assigned), the problem not exist.
  qemu-cmds:
/usr/libexec/qemu-kvm -name rhel7 \
-machine q35,accel=kvm,usb=off,vmport=off \
-cpu SandyBridge \
-m 10240 \
-realtime mlock=off \
-smp 3 \
-uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
-no-user-config -nodefaults \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
-no-hpet \
-boot menu=on,splash-time=12000 \
-device vfio-pci,host=03:00.0,id=GPU1 \
-vga qxl \
-monitor stdio \
-vnc 0:2 \
-netdev tap,id=hostnet10 \
-device virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
-spice ipv4,port=5000,disable-ticketing \
-enable-kvm \
-global ICH9-LPC.disable_s3=0 \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
-drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
-device ahci,id=ahci \
-drive file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,format=qcow2,cache=none,aio=native \
-device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
-global driver=cfi.pflash01,property=secure,value=on \
-chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
-device isa-debugcon,chardev=ovmf_log,iobase=0x402 \

Test environment is ok now, retest it.

Test Versions:
 kernel version: 3.10.0-504.el7.x86_64
 qemu-kvm-rhev version: qemu-kvm-rhev-2.6.0-22.el7.x86_64
 OVMF version: OVMF-20160608-3.git988715a.el7.noarch
 Seabios version: seabios-bin-1.9.1-4.el7.noarch
 
Test Steps:
1.Get GPU Devices:
#lspci
.....
03:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
.....

2.Get GPU Device Vendor/Device ID.
# lspci -n -s 04:00.0
04:00.0 0300: 10de:11ba (rev a1)
# lspci -n -s 04:00.1
04:00.1 0403: 10de:0e0a (rev a1)
# lspci -n -s 03:00.0
03:00.0 0300: 10de:11ba (rev a1)
# lspci -n -s 03:00.1
03:00.1 0403: 10de:0e0a (rev a1)

3.Add GPU Device Vendor/Device ID to cmdline.
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-504.el7.x86_64 root=/dev/mapper/rhel_dhcp--65--197-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-65-197/root rd.lvm.lv=rhel_dhcp-65-197/swap rhgb quiet LANG=en_US.UTF-8 intel_iommu=on pci-stub.ids=10de:11ba,10de:0e0a

4.Load vfio/vfio-pci module.
#modprobe vfio/vfio-pci

5.Unbind GPU Device from host.
#echo 0000:04:00.0 > /sys/bus/pci/devices/0000\:04\:00.0/driver/unbind 
#echo 0000:04:00.1 > /sys/bus/pci/devices/0000\:04\:00.1/driver/unbind

6.Assign GPU Device to vfio-pci
# echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/new_id 
# echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/remove_id 
# echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/new_id
# echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/remove_id 

5.Start guest with GPU device(VGA Controller+Audio Device):
/usr/libexec/qemu-kvm -name rhel7 \
-machine q35,accel=kvm,usb=off,vmport=off \
-cpu host \
-m 10240 \
-realtime mlock=off \
-smp 3 \
-uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
-no-user-config -nodefaults \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
-mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
-no-hpet \
-boot menu=on,splash-time=12000 \
-device vfio-pci,host=04:00.0,id=GPU1 \
-device vfio-pci,host=04:00.1,id=GPU2 \
-vga qxl \
-monitor stdio \
-vnc 0:2 \
-netdev tap,id=hostnet10 \
-device virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
-spice ipv4,port=5000,disable-ticketing \
-enable-kvm \
-global ICH9-LPC.disable_s3=0 \
-drive file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
-drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
-device ahci,id=ahci \
-drive file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,format=qcow2,cache=none,aio=native \
-device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
-global driver=cfi.pflash01,property=secure,value=on \
-chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
-device isa-debugcon,chardev=ovmf_log,iobase=0x402 \

Test Result:
 Screen still show black.

Logs as attachment：
 ovmf_pci_stub.log

Additional info:
 When do the same test with seabios, the problem not exist.

Created attachment 1202693 [details]
ovmf_pci_stub.log

(In reply to aihua liang from comment #37)
> Test Environment with K5000 was destroyed, so can't give more info now.
> 
> Additional Info:
> 1. When test with K2(no audio device), the problem not exist.
>   qemu-cmds:
> /usr/libexec/qemu-kvm -name rhel7 \
> -machine q35 \
> -cpu host \
> -m 10240 \
> -realtime mlock=off \
> -smp 3 \
> -uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
> -no-user-config -nodefaults \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
> -no-hpet \
> -boot menu=on,splash-time=12000 \
> -vga qxl \
> -monitor stdio \
> -vnc 0:2 \
> -netdev tap,id=hostnet1 \
> -device virtio-net-pci,netdev=hostnet1,id=net1,mac=58:54:00:49:b2:5f \
> -spice ipv4,port=5000,disable-ticketing \
> -enable-kvm \
> -global ICH9-LPC.disable_s3=0 \
> -drive
> file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
> -drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
> -device ahci,id=ahci \
> -drive
> file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,
> format=qcow2,cache=none,aio=native \
> -device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
> -global driver=cfi.pflash01,property=secure,value=on \
> -chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
> -device isa-debugcon,chardev=ovmf_log,iobase=0x402 \
> -device vfio-pci,host=84:00.0,id=GPU1 \
> 
> 
2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio Controller assigned), the problem also exist.
   qemu-cmds:
 /usr/libexec/qemu-kvm -name rhel7 \
 -machine q35,accel=kvm,usb=off,vmport=off \
 -cpu SandyBridge \
 -m 10240 \
 -realtime mlock=off \
 -smp 3 \
 -uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
 -no-user-config -nodefaults \
 -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
 -mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
 -no-hpet \
 -boot menu=on,splash-time=12000 \
 -device vfio-pci,host=03:00.0,id=GPU1 \
 -vga qxl \
 -monitor stdio \
 -vnc 0:2 \
 -netdev tap,id=hostnet10 \
 -device
 virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
 -spice ipv4,port=5000,disable-ticketing \
 -enable-kvm \
 -global ICH9-LPC.disable_s3=0 \
 -drive
 file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
 -drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
 -device ahci,id=ahci \
 -drive
 file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,
 format=qcow2,cache=none,aio=native \
 -device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
 -global driver=cfi.pflash01,property=secure,value=on \
 -chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
 -device isa-debugcon,chardev=ovmf_log,iobase=0x402 \

(In reply to aihua liang from comment #39)
> Created attachment 1202693 [details]
> ovmf_pci_stub.log

Yeah it's the exact same issue as before. Bugged out GOP driver in the card's UEFI oprom image.

(In reply to aihua liang from comment #37)
> 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> Controller assigned), the problem not exist.
>   qemu-cmds:
> /usr/libexec/qemu-kvm -name rhel7 \
> -machine q35,accel=kvm,usb=off,vmport=off \
> -cpu SandyBridge \
> -m 10240 \
> -realtime mlock=off \
> -smp 3 \
> -uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
> -no-user-config -nodefaults \
> -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
> -mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
> -no-hpet \
> -boot menu=on,splash-time=12000 \
> -device vfio-pci,host=03:00.0,id=GPU1 \
> -vga qxl \
> -monitor stdio \
> -vnc 0:2 \
> -netdev tap,id=hostnet10 \
> -device
> virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
> -spice ipv4,port=5000,disable-ticketing \
> -enable-kvm \
> -global ICH9-LPC.disable_s3=0 \
> -drive
> file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
> -drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
> -device ahci,id=ahci \
> -drive
> file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,
> format=qcow2,cache=none,aio=native \
> -device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
> -global driver=cfi.pflash01,property=secure,value=on \
> -chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
> -device isa-debugcon,chardev=ovmf_log,iobase=0x402 \

(In reply to aihua liang from comment #40)
> 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> Controller assigned), the problem also exist.
>    qemu-cmds:
>  /usr/libexec/qemu-kvm -name rhel7 \
>  -machine q35,accel=kvm,usb=off,vmport=off \
>  -cpu SandyBridge \
>  -m 10240 \
>  -realtime mlock=off \
>  -smp 3 \
>  -uuid 1534fa42-4818-4493-9f67-eee5ba758385 \
>  -no-user-config -nodefaults \
>  -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/test1,server,nowait \
>  -mon chardev=qmp_id_catch_monitor,id=monitor,mode=control \
>  -no-hpet \
>  -boot menu=on,splash-time=12000 \
>  -device vfio-pci,host=03:00.0,id=GPU1 \
>  -vga qxl \
>  -monitor stdio \
>  -vnc 0:2 \
>  -netdev tap,id=hostnet10 \
>  -device
>  virtio-net-pci,netdev=hostnet10,id=net10,mac=58:54:00:49:b2:5f,bootindex=1 \
>  -spice ipv4,port=5000,disable-ticketing \
>  -enable-kvm \
>  -global ICH9-LPC.disable_s3=0 \
>  -drive
>  file=/usr/share/OVMF/OVMF_CODE.secboot.fd,if=pflash,format=raw,readonly=on \
>  -drive file=/usr/share/OVMF/OVMF_Client_VARS.fd,if=pflash,format=raw \
>  -device ahci,id=ahci \
>  -drive
>  file=/home/rhel72_secure.qcow2,if=none,id=drive-ide-3,media=disk,
>  format=qcow2,cache=none,aio=native \
>  -device ide-drive,drive=drive-ide-3,id=ide3,bus=ahci.0 \
>  -global driver=cfi.pflash01,property=secure,value=on \
>  -chardev socket,id=ovmf_log,path=/var/tmp/ovmf_log,server,nowait \
>  -device isa-debugcon,chardev=ovmf_log,iobase=0x402 \

What's the difference between these?  Did the host environment change?  How?

(In reply to aihua liang from comment #38)
> Test environment is ok now, retest it.

How is the host configured?  Document it.

> What's the difference between these?  Did the host environment change?  How?
 
Hi,Alex

  Yeath, K2 is on another host in beaker.

Hi,zhguo
  Can you give the difference between k2 host and k5000 host?
  BTW,test step on K2 & K5000 is almost the same.
    (except:Need to ubind the audio device with K5000 VGA Controller together)

> (In reply to aihua liang from comment #38)
> > Test environment is ok now, retest it.
> 
> How is the host configured?  Document it.

 1.Add intel_iommu=on & pci-stub.ids=10de:11ba,10de:0e0a in host cmdline(/boot/grub2/grub.cfg),reboot host.

 2.When it boot up,check iommu_group
# find /sys/kernel/iommu_groups/ -type l | grep 04
/sys/kernel/iommu_groups/30/devices/0000:04:00.0
/sys/kernel/iommu_groups/30/devices/0000:04:00.1

 3.Check if pci-stub take effect.
#ll /sys/bus/pci/devices/0000\:04\:00.0/driver
lrwxrwxrwx. 1 root root 0 Sep 21 15:24 /sys/bus/pci/devices/0000:04:00.0/driver -> ../../../../bus/pci/drivers/pci-stub

#ll /sys/bus/pci/devices/0000\:04\:00.1/driver
lrwxrwxrwx. 1 root root 0 Sep 21 15:28 /sys/bus/pci/devices/0000:04:00.1/driver -> ../../../../bus/pci/drivers/pci-stub

 4.Load vfio-pci--modprobe vfio-pci.
# lsmod | grep vfio
vfio_pci               36948  0 
vfio_iommu_type1       17632  0 
vfio                   26136  2 vfio_iommu_type1,vfio_pci
irqbypass              13503  2 kvm,vfio_pci

 5.Unbind device from pci-stub driver,and check it.
# echo 0000:04:00.0 > /sys/bus/pci/devices/0000\:04\:00.0/driver/unbind 
# echo 0000:04:00.1 > /sys/bus/pci/devices/0000\:04\:00.1/driver/unbind 
# ll /sys/bus/pci/devices/0000\:04\:00.0/driver
ls: cannot access /sys/bus/pci/devices/0000:04:00.0/driver: No such file or directory
# ll /sys/bus/pci/devices/0000\:04\:00.1/driver
ls: cannot access /sys/bus/pci/devices/0000:04:00.1/driver: No such file or directory

 6. Bind device to vfio-pci driver, check it.
# echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/new_id 
# echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/remove_id 
# echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/new_id 
# echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/remove_id

# ll /sys/bus/pci/devices/0000\:04\:00.0/driver
lrwxrwxrwx. 1 root root 0 Sep 21 15:44 /sys/bus/pci/devices/0000:04:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci

# ll /sys/bus/pci/devices/0000\:04\:00.1/driver
lrwxrwxrwx. 1 root root 0 Sep 21 15:44 /sys/bus/pci/devices/0000:04:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci

 7.Start guest with gpu device.
   /usr/libexec/qemu-kvm ....-device vfio-pci,host=04:00.0,id=GPU1 -device vfio-pci,host=04:00.1,id=GPU2 ...

 Is the above message what you need? 
 If not, please give me an example of your needed host configuration, or you can ask zhiyi Guo to provide more.  


Thanks,
aliang

(In reply to aihua liang from comment #43)
> > What's the difference between these?  Did the host environment change?  How?
>  
> Hi,Alex
> 
>   Yeath, K2 is on another host in beaker.

(In reply to Alex Williamson from comment #42)
> (In reply to aihua liang from comment #37)
> > 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> > Controller assigned), the problem not exist.

K5000...

> (In reply to aihua liang from comment #40)
> > 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> > Controller assigned), the problem also exist.

K5000...

(In reply to aihua liang from comment #43)
> > (In reply to aihua liang from comment #38)
> > > Test environment is ok now, retest it.
> > 
> > How is the host configured?  Document it.
> 
>  1.Add intel_iommu=on & pci-stub.ids=10de:11ba,10de:0e0a in host
> cmdline(/boot/grub2/grub.cfg),reboot host.
> 
>  2.When it boot up,check iommu_group
> # find /sys/kernel/iommu_groups/ -type l | grep 04
> /sys/kernel/iommu_groups/30/devices/0000:04:00.0
> /sys/kernel/iommu_groups/30/devices/0000:04:00.1
> 
>  3.Check if pci-stub take effect.
> #ll /sys/bus/pci/devices/0000\:04\:00.0/driver
> lrwxrwxrwx. 1 root root 0 Sep 21 15:24
> /sys/bus/pci/devices/0000:04:00.0/driver ->
> ../../../../bus/pci/drivers/pci-stub
> 
> #ll /sys/bus/pci/devices/0000\:04\:00.1/driver
> lrwxrwxrwx. 1 root root 0 Sep 21 15:28
> /sys/bus/pci/devices/0000:04:00.1/driver ->
> ../../../../bus/pci/drivers/pci-stub
> 
>  4.Load vfio-pci--modprobe vfio-pci.
> # lsmod | grep vfio
> vfio_pci               36948  0 
> vfio_iommu_type1       17632  0 
> vfio                   26136  2 vfio_iommu_type1,vfio_pci
> irqbypass              13503  2 kvm,vfio_pci
> 
>  5.Unbind device from pci-stub driver,and check it.
> # echo 0000:04:00.0 > /sys/bus/pci/devices/0000\:04\:00.0/driver/unbind 
> # echo 0000:04:00.1 > /sys/bus/pci/devices/0000\:04\:00.1/driver/unbind 
> # ll /sys/bus/pci/devices/0000\:04\:00.0/driver
> ls: cannot access /sys/bus/pci/devices/0000:04:00.0/driver: No such file or
> directory
> # ll /sys/bus/pci/devices/0000\:04\:00.1/driver
> ls: cannot access /sys/bus/pci/devices/0000:04:00.1/driver: No such file or
> directory
> 
>  6. Bind device to vfio-pci driver, check it.
> # echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/new_id 
> # echo "10de 11ba" >/sys/bus/pci/drivers/vfio-pci/remove_id 
> # echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/new_id 
> # echo "10de 0e0a" >/sys/bus/pci/drivers/vfio-pci/remove_id
> 
> # ll /sys/bus/pci/devices/0000\:04\:00.0/driver
> lrwxrwxrwx. 1 root root 0 Sep 21 15:44
> /sys/bus/pci/devices/0000:04:00.0/driver ->
> ../../../../bus/pci/drivers/vfio-pci
> 
> # ll /sys/bus/pci/devices/0000\:04\:00.1/driver
> lrwxrwxrwx. 1 root root 0 Sep 21 15:44
> /sys/bus/pci/devices/0000:04:00.1/driver ->
> ../../../../bus/pci/drivers/vfio-pci
> 
>  7.Start guest with gpu device.
>    /usr/libexec/qemu-kvm ....-device vfio-pci,host=04:00.0,id=GPU1 -device
> vfio-pci,host=04:00.1,id=GPU2 ...
> 
>  Is the above message what you need? 
>  If not, please give me an example of your needed host configuration, or you
> can ask zhiyi Guo to provide more.  

I was more asking about the issues we worked through in comments 28 through 36, how many GPUs are installed in the system?  What are they?  Is the assigned GPU the boot VGA device?

(In reply to Alex Williamson from comment #46)


***************Section one*************** 
> (In reply to aihua liang from comment #43)

***************Section one**************
> > > (In reply to aihua liang from comment #38)
> > > > Test environment is ok now, retest it.
> > > 
> > > How is the host configured?  Document it.

> I was more asking about the issues we worked through in comments 28 through
> 36, how many GPUs are installed in the system?  What are they?  Is the
> assigned GPU the boot VGA device?

I run the test on Zhiyi Guo's test environment, so the host configurations as bellow:
 
When run test with K5000 on workstation...
  1)how many GPUs are installed in the system? ------3 GPUs.

  2)What are they?
# lspci | grep VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [Quadro K5000] (rev a1)
07:00.0 VGA compatible controller: NVIDIA Corporation GF108GL [Quadro 600] (rev a1)

  3)Is the assigned GPU the boot VGA device? ---No
      boot VGA device:Quadro 600
      Assigned GPU:Quadro K5000

 When run test with K2 on server...
   1)how many GPUs are installed in the system? ------3 GPUs.

   2)What are they?
# lspci | grep VGA
06:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
07:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
08:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [GRID K1] (rev a1)
10:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)
84:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)
85:00.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K2] (rev a1)

  3)Is the assigned GPU the boot VGA device? ---No 
       boot VGA device: None
       Assigned GPU: GRID K2


*****************Section two ******************
> > (In reply to aihua liang from comment #37)
> > > 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> > > Controller assigned), the problem not exist.
>K5000...

I gave the wrong summary by mistake. I found the mistake some day, wanted to correct it, but the comment can't be edited, so i renewed the summary in comment #40, but forgot to give a statement for the correction.

Sorry for confusing.


>> (In reply to aihua liang from comment #40)
> > > 2.When test with K5000 assigned to VM(no NVIDIA Corporation GK104 HDMI Audio
> > >Controller assigned), the problem also exist.

>K5000...



Thanks,
aliang

(In reply to aihua liang from comment #49)

Correct boot VGA info when run test on server with K2.

>  When run test with K2 on server...
> 
>   3)Is the assigned GPU the boot VGA device? ---No 
>        boot VGA device: None 

         boot VGA device: Matrox Electronics Systems Ltd. G200eR2

>        Assigned GPU: GRID K2

Note that the libvirt syntax we'll want to document is
whatever ends up being implemented to solve Bug 1425058,
so I'm adding a dependency on it.

Andrea, could you briefly explain (or link a good source about) what changes were actually applied in terms of this feature, so I can decide how best to approach the documentation? I tried looking into BZ 1425058 but couldn't quite figure out what the user impact of the changes would be...

Thanks!
J.

(In reply to Jiri Herrmann from comment #58)
> Andrea, could you briefly explain (or link a good source about) what changes
> were actually applied in terms of this feature, so I can decide how best to
> approach the documentation? I tried looking into BZ 1425058 but couldn't
> quite figure out what the user impact of the changes would be...

Hopefully

  https://bugzilla.redhat.com/show_bug.cgi?id=1425058#c34

is the brief explanation you're looking for; a more detailed one
can be found in

  https://bugzilla.redhat.com/show_bug.cgi?id=1425058#c7

From the user's point of view, the newly introduced configuration
option

  <rom enable='no'/>

provides a way to disable PCI ROMs altogether for a device: that
could also be achieved in the past using

  <rom file=''/>

but only managed to do so by abusing the intended semantics for the
file attribute and relying on a quirk of QEMU, whereas the new one
is specifically intended to disable PCI ROMs and could be easily
implemented for other hypervisors.

Another form that for the most part worked, but did not actually
disable all possible ways QEMU would attempt to load a PCI ROM, was

  <rom bar='off'/>

So if users were relying on the old variants for the purpose of
disabling PCI ROMs, they should switch to the new one; and as far
as documentation is concerned, the new syntax is the one we should
point users to (see Comment 11 and Comment 56).

Hope this clears things up! :)

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days