Bug 1918113

Summary:	Windows guest bsod with win8(q35) after install viogpudo driver
Product:	Red Hat Enterprise Linux 8	Reporter:	dehanmeng <demeng>
Component:	virtio-win	Assignee:	Vadim Rozenfeld <vrozenfe>
virtio-win sub component:	virtio-win-prewhql	QA Contact:	dehanmeng <demeng>
Status:	CLOSED CURRENTRELEASE	Docs Contact:
Severity:	high
Priority:	high	CC:	ailan, junzhang, lijin, mdean, vrozenfe
Version:	8.4	Keywords:	Triaged
Target Milestone:	rc
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-16 09:40:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1936265
Bug Blocks:	1948357

Description dehanmeng 2021-01-20 05:32:43 UTC

Description of problem:
Hit bsod situation when run case https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL-196499 manually. 

Version-Release number of selected component (if applicable):
qemu-kvm-4.2.0-40.module+el8.4.0+9278+dd53883d.x86_64
virtio-win-prewhql-193
kernel-4.18.0-270.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot up guest
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -m 13312  \
    -device virtio-vga,id=video0 \                                               
    -smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0xfff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,path=/tmp/avocado_9ms7xm0w/monitor-qmpmonitor1-20210105-032815-BLpNoZnG,id=qmp_id_qmpmonitor1,nowait,server  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/tmp/avocado_9ms7xm0w/monitor-catch_monitor-20210105-032815-BLpNoZnG,id=qmp_id_catch_monitor,nowait,server  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idXAAXfN \
    -chardev socket,path=/tmp/avocado_9ms7xm0w/serial-serial0-20210105-032815-BLpNoZnG,id=chardev_serial0,nowait,server \
    -device isa-serial,id=serial0,chardev=chardev_serial0 \
    -chardev socket,path=/tmp/avocado_9ms7xm0w/serial-org.qemu.guest_agent.0-20210105-032815-BLpNoZnG,id=chardev_org.qemu.guest_agent.0,nowait,server \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device virtio-serial-pci,id=virtio_serial_pci0,bus=pcie-root-port-1,addr=0x0 \
    -device virtserialport,id=org.qemu.guest_agent.0,name=org.qemu.guest_agent.0,chardev=chardev_org.qemu.guest_agent.0,bus=virtio_serial_pci0.0,nr=1  \
    -chardev socket,id=seabioslog_id_20210105-032815-BLpNoZnG,path=/tmp/avocado_9ms7xm0w/seabios-20210105-032815-BLpNoZnG,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20210105-032815-BLpNoZnG,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-2,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-3,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=win8-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:53:3e:06:b0:47,id=idKpQvyY,netdev=idpFCo1s,bus=pcie-root-port-4,addr=0x0  \
    -netdev tap,id=idpFCo1s,vhost=on \
    -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \
    -blockdev node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=virtio-win-prewhql-0.1-193.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_virtio \
    -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on  \
    -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \   
    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \  
    -vnc :1  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
2.as polarion case step


Actual results:
guest hit bsod, stop code: SYSTEM_THREAD_EXCEPTION_NOT_HANDLED viogpudo.sys

Expected results:
viogpudo driver can be installed successfully and vga function works well.

Additional info:
Dump file are listed in http://fileshare.englab.nay.redhat.com/pub/logs/install_virtio-gpu_memory_dmp/

Comment 9 Vadim Rozenfeld 2021-02-02 06:09:25 UTC

Can you please give a try to drivers from build 194
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1484340 ?

Comment 10 dehanmeng 2021-02-02 14:46:30 UTC

(In reply to Vadim Rozenfeld from comment #9)
> Can you please give a try to drivers from build 194
> https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1484340 ?

Hi Vadim,
Thanks for your effort,Vadim. In brief, I tried lots of time on win8-like guest with several driver combinations.
[1] the whole screen is black and couldn't do anything after installed viogpudo driver.
   -device virtio-vga,id=video0 \ 
   -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
   -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
[2] result like previous situation.
   -device VGA,bus=pcie.0,addr=0x2 \
   -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
   -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
[3] guest act like normal if I just add virtio-vga directly, mouse keyboard and cursor can work well. looks good.
   -device virtio-vga,id=video0 \ 
but like we talked before, the best combination is wrap virtio-vga with viogpudo driver. so what do you think about it? 

BRs
Dehan

Comment 11 Vadim Rozenfeld 2021-02-02 20:36:36 UTC

(In reply to dehanmeng from comment #10)
> (In reply to Vadim Rozenfeld from comment #9)
> > Can you please give a try to drivers from build 194
> > https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1484340 ?
> 
> Hi Vadim,
> Thanks for your effort,Vadim. In brief, I tried lots of time on win8-like
> guest with several driver combinations.
> [1] the whole screen is black and couldn't do anything after installed
> viogpudo driver.
>    -device virtio-vga,id=video0 \ 
>    -device
> pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> chassis=6 \
>    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> [2] result like previous situation.
>    -device VGA,bus=pcie.0,addr=0x2 \
>    -device
> pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> chassis=6 \
>    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> [3] guest act like normal if I just add virtio-vga directly, mouse keyboard
> and cursor can work well. looks good.
>    -device virtio-vga,id=video0 \ 
> but like we talked before, the best combination is wrap virtio-vga with
> viogpudo driver. so what do you think about it? 
> 


Hi Dehan,

You need to install viogpudo driver not for virtio-vga device only
but for virtio-gpu-pci device as well.

Cheers,
Vadim.



> BRs
> Dehan

Comment 12 dehanmeng 2021-02-05 01:41:33 UTC

(In reply to Vadim Rozenfeld from comment #11)
> (In reply to dehanmeng from comment #10)
> > (In reply to Vadim Rozenfeld from comment #9)
> > > Can you please give a try to drivers from build 194
> > > https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1484340 ?
> > 
> > Hi Vadim,
> > Thanks for your effort,Vadim. In brief, I tried lots of time on win8-like
> > guest with several driver combinations.
> > [1] the whole screen is black and couldn't do anything after installed
> > viogpudo driver.
> >    -device virtio-vga,id=video0 \ 
> >    -device
> > pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> > chassis=6 \
> >    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> > [2] result like previous situation.
> >    -device VGA,bus=pcie.0,addr=0x2 \
> >    -device
> > pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> > chassis=6 \
> >    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> > 
> 
> 
> Hi Dehan,
> 
> You need to install viogpudo driver not for virtio-vga device only
> but for virtio-gpu-pci device as well.
> 
so as you said both of the combination I've tested but the screen turn to black, maybe you also need to take effort for it, Vadim. could you?

Comment 13 Vadim Rozenfeld 2021-02-05 02:31:35 UTC

(In reply to dehanmeng from comment #12)
> (In reply to Vadim Rozenfeld from comment #11)
> > (In reply to dehanmeng from comment #10)
> > > (In reply to Vadim Rozenfeld from comment #9)
> > > > Can you please give a try to drivers from build 194
> > > > https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1484340 ?
> > > 
> > > Hi Vadim,
> > > Thanks for your effort,Vadim. In brief, I tried lots of time on win8-like
> > > guest with several driver combinations.
> > > [1] the whole screen is black and couldn't do anything after installed
> > > viogpudo driver.
> > >    -device virtio-vga,id=video0 \ 
> > >    -device
> > > pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> > > chassis=6 \
> > >    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> > > [2] result like previous situation.
> > >    -device VGA,bus=pcie.0,addr=0x2 \
> > >    -device
> > > pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x1.0x5,bus=pcie.0,
> > > chassis=6 \
> > >    -device virtio-gpu-pci,id=video1,bus=pcie-root-port-5,addr=0x0 \
> > > 
> > 
> > 
> > Hi Dehan,
> > 
> > You need to install viogpudo driver not for virtio-vga device only
> > but for virtio-gpu-pci device as well.
> > 
> so as you said both of the combination I've tested but the screen turn to
> black, maybe you also need to take effort for it, Vadim. could you?

Sure, let me reproduce the problem on my setup.
Best,
Vadim.

Comment 14 dehanmeng 2021-03-03 07:17:34 UTC

Hi Vadim,
Hit bsod issue on win2016-64 guest recently when I run "shutdown" operation case. guest was booted up with virtio-win-prewhql-196.iso and just one gpu driver "-device virtio-win vga,id=video0 \". but I didn't hit a few days before. actually I just re-install my machine, anyway, this is more urgent situation for now. I'm also testing other win10-like and win8-like guests. will update others info here once I get new data.

packages:
qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb.x86_64
kernel-4.18.0-291.el8.x86_64
seabios-1.14.0-1.module+el8.4.0+8855+a9e237a9.x86_64
RHEL-8.4.0-20210223.n.0

step to reproduce:
1. boot up guest with following qemu-command line in link[1]:

2. install driver :
virtio-gpu driver from virtio-win.iso during guest installation:
during the installation stage click "load driver "-->"which driver you want to choose" click browse, choose the driver we need .
3.Enable viogpudo.sys in cmd
  1).For win10,win2016,win2019: verifier /flags 0x009209BB /driver viogpudo.sys
  2). verifier /querysettings
4.Shutdown guest through qmp monitor:
  1) [root@dell-per440-06 Bug_1861229_WDDM_virtio_vga]# nc -U /tmp/avocado_adfijf889/monitor-qmpmonitor1-20210105-032815-BLpNoZnG
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 2, "major": 5}, "package": "qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb"}, "capabilities": ["oob"]}}
{"execute":"qmp_capabilities"}
{"return": {}}
{"execute":"system_powerdown"}
{"return": {}}
{"timestamp": {"seconds": 1614772460, "microseconds": 98975}, "event": "POWERDOWN"}
{"timestamp": {"seconds": 1614772464, "microseconds": 283704}, "event": "VSERPORT_CHANGE", "data": {"open": false, "id": "org.qemu.guest_agent.0"}}
{"timestamp": {"seconds": 1614772469, "microseconds": 299277}, "event": "SHUTDOWN", "data": {"guest": false, "reason": "host-signal"}}

and then the actual result is the guest hit bsod and collecting dmp file. please check script and dmp file below link.
[1] http://fileshare.englab.nay.redhat.com/pub/logs/196GPU_functional_test_result/

Comment 15 dehanmeng 2021-03-03 07:22:07 UTC

(In reply to dehanmeng from comment #14)
> Hi Vadim,
> Hit bsod issue on win2016-64 guest recently when I run "shutdown" operation
> case. guest was booted up with virtio-win-prewhql-196.iso and just one gpu
> driver "-device virtio-win vga,id=video0 \". but I didn't hit a few days
> before. actually I just re-install my machine, anyway, this is more urgent
> situation for now. I'm also testing other win10-like and win8-like guests.
> will update others info here once I get new data.
> 
> packages:
> qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb.x86_64
> kernel-4.18.0-291.el8.x86_64
> seabios-1.14.0-1.module+el8.4.0+8855+a9e237a9.x86_64
> RHEL-8.4.0-20210223.n.0
> 
> step to reproduce:
> 1. boot up guest with following qemu-command line in link[1]:
> 
> 2. install driver :
> virtio-gpu driver from virtio-win.iso during guest installation:
> during the installation stage click "load driver "-->"which driver you want
> to choose" click browse, choose the driver we need .
> 3.Enable viogpudo.sys in cmd
>   1).For win10,win2016,win2019: verifier /flags 0x009209BB /driver
> viogpudo.sys
>   2). verifier /querysettings
> 4.Shutdown guest through qmp monitor:
>   1) [root@dell-per440-06 Bug_1861229_WDDM_virtio_vga]# nc -U
> /tmp/avocado_adfijf889/monitor-qmpmonitor1-20210105-032815-BLpNoZnG
> {"QMP": {"version": {"qemu": {"micro": 0, "minor": 2, "major": 5},
> "package": "qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb"},
> "capabilities": ["oob"]}}
> {"execute":"qmp_capabilities"}
> {"return": {}}
> {"execute":"system_powerdown"}
> {"return": {}}
> {"timestamp": {"seconds": 1614772460, "microseconds": 98975}, "event":
> "POWERDOWN"}
> {"timestamp": {"seconds": 1614772464, "microseconds": 283704}, "event":
> "VSERPORT_CHANGE", "data": {"open": false, "id": "org.qemu.guest_agent.0"}}
> {"timestamp": {"seconds": 1614772469, "microseconds": 299277}, "event":
> "SHUTDOWN", "data": {"guest": false, "reason": "host-signal"}}
> 
> and then the actual result is the guest hit bsod and collecting dmp file.
> please check script and dmp file below link.
> [1]
> http://fileshare.englab.nay.redhat.com/pub/logs/
> 196GPU_functional_test_result/

the stop code is : DRIVER IRQL NOT LESS OR EQUAL

Comment 16 dehanmeng 2021-03-03 07:56:10 UTC

(In reply to dehanmeng from comment #15)
> (In reply to dehanmeng from comment #14)
> > Hi Vadim,
> > Hit bsod issue on win2016-64 guest recently when I run "shutdown" operation
> > case. guest was booted up with virtio-win-prewhql-196.iso and just one gpu
> > driver "-device virtio-win vga,id=video0 \". but I didn't hit a few days
> > before. actually I just re-install my machine, anyway, this is more urgent
> > situation for now. I'm also testing other win10-like and win8-like guests.
> > will update others info here once I get new data.
> > 
> > packages:
> > qemu-kvm-5.2.0-8.module+el8.4.0+10093+e085f1eb.x86_64
    kernel-4.18.0-287.el8.dt4.x86_64
> > seabios-1.14.0-1.module+el8.4.0+8855+a9e237a9.x86_64
    RHEL-8.4.0-20210223.d.2
> > 
for win2019/win10-64, I guest it might be related my machine, I'm testing gpu on other machine.will update result soon.

Comment 17 Vadim Rozenfeld 2021-03-03 09:35:03 UTC

Caused by kernel synchronization delay fuzzing flag https://bugzilla.redhat.com/show_bug.cgi?id=1826352
Not a blocker. Problem will be fixed in 8.5

Comment 20 Vadim Rozenfeld 2021-05-11 08:26:48 UTC

https://github.com/virtio-win/kvm-guest-drivers-windows/pull/577

Comment 21 Vadim Rozenfeld 2021-05-17 00:16:25 UTC

Can we give a try to build 199
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1605219

Comment 22 dehanmeng 2021-05-18 04:21:50 UTC

reproduce this issue:
version:
RHEL-8.5.0-20210428.d.3
virtio-win-prewhql-0.1-193.iso
seabios-1.14.0-1.scrmod+el8.5.0+10801+f1aef2c6.x86_64
kernel-4.18.0-304.5.el8.kpq1.x86_64
qemu-kvm-6.0.0-15rc5.scrmod+el8.5.0+10801+f1aef2c6.wrb210428.x86_64

steps:
1.boot up win8.1-64/32/win2012-64/2012r2-64 guests
2. install viogpudo driver on device manager

Actual result:
Four guest get bosd situation.

Expected result:
Driver install successfully and display function/basic interaction should be fine.

Reproduce this issue:
version:
RHEL-8.5.0-20210428.d.3
virtio-win-prewhql-0.1-193.iso
seabios-1.14.0-1.scrmod+el8.5.0+10801+f1aef2c6.x86_64
kernel-4.18.0-304.5.el8.kpq1.x86_64
qemu-kvm-6.0.0-15rc5.scrmod+el8.5.0+10801+f1aef2c6.wrb210428.x86_64

steps as above

Actual result:
Driver install successfully and display function/basic interaction should be fine.

Expected result:
Driver install successfully and display function/basic interaction should be fine.

Thanks for your effort Vadim. win8+ bsod issue is gone.

Cheers
Dehan Meng

Comment 23 Vadim Rozenfeld 2021-05-18 07:10:07 UTC

Thanks a lot for the update Dehan,

Just one thing I would like to confirm with you. Did you check that
the driver works fine with delay fuzzing flag enabled?

Best,
Vadim.

Comment 24 dehanmeng 2021-05-18 09:23:20 UTC

(In reply to Vadim Rozenfeld from comment #23)
> Thanks a lot for the update Dehan,
> 
> Just one thing I would like to confirm with you. Did you check that
> the driver works fine with delay fuzzing flag enabled?
> 
> Best,
> Vadim.

Hi Vadim,

I've tested the thing you said, run 'verifier /flags 0x008209BB /driver viogpudo.sys' on win8+ guests, All of them can run well that display functions and click anywhere after that operations. but win2012-64/win8.1-64 couldn't reboot/shutdown immediately and would hit black screen after shutdown/reboot commands were operated. maybe it just needs more time to shutdown/reboot. I'll run it again and wait for it. if the black-screen situation always be there. I'll update more details here. 

Thanks
Dehan Meng