Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1872384

Summary: [NetKVM] BSOD in Sleep during IO test with HCK 1809
Product: Red Hat Enterprise Linux 8 Reporter: ybendito
Component: virtio-winAssignee: ybendito
virtio-win sub component: virtio-win-prewhql QA Contact: Peixiu Hou <phou>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: kanderso, lijin, mdean, phou, vrozenfe, xiagao, ybendito, yvugenfi
Version: 8.4Flags: pm-rhel: mirror+
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-16 14:24:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Dump file
none
Symbol file
none
bsod screenshot of win832 none

Description ybendito 2020-08-25 15:54:35 UTC 
Created attachment 1712566 [details]
Dump file

Created attachment 1712566 [details]
Dump file

Description of problem:
BSOD in Sleep during IO test with HCK 1809
Dump and symbol file attached

Version-Release number of selected component (if applicable):
0.1.189

How reproducible:
At least once

Steps to Reproduce:
Specified HCK test, found during HCK-CI
Comment 1 ybendito 2020-08-25 15:55:50 UTC 
Created attachment 1712568 [details]
Symbol file
Comment 2 xiagao 2020-08-26 02:20:24 UTC 
Hi,
I tried to reproduce this bug.
Could you tell which case hit this issue?And your host machine is amd or intel?

The days before I used hlk-1809 to test win2019 nekvm device(driver:189),but didn't hit this issue.My host cpu is intel.

Thanks,
Xiaoling
Comment 3 xiagao 2020-08-26 02:23:55 UTC 
(In reply to xiagao from comment #2)
> Hi,
> I tried to reproduce this bug.
> Could you tell which case hit this issue?And your host machine is amd or
> intel?
> 
> The days before I used hlk-1809 to test win2019 nekvm device(driver:189),but
> didn't hit this issue.My host cpu is intel.
> 
> Thanks,
> Xiaoling

The test case is "sleep with IO during reliability", I found it.
Comment 4 xiagao 2020-08-26 06:13:50 UTC 
I run win2019 guest in hlk-1809 with 189 driver for 16 times, all pass, didn't reproduce this issue.
Comment 5 xiagao 2020-08-26 07:03:38 UTC 
I run win832 in hck, hit BSOD.
Could you help check if they are the same issue?

DUMP file is in below link.
http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/netkvm-dump/dump-win832.tar.gz

It happened during running "Configure Driver Verifier for Device Test"
Comment 6 xiagao 2020-08-26 07:34:24 UTC 
Created attachment 1712638 [details]
bsod screenshot of win832
Comment 7 Yvugenfi@redhat.com 2020-08-26 09:50:52 UTC 
(In reply to xiagao from comment #5)
> I run win832 in hck, hit BSOD.
> Could you help check if they are the same issue?
> 
> DUMP file is in below link.
> http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/
> netkvm-dump/dump-win832.tar.gz
> 
> It happened during running "Configure Driver Verifier for Device Test"

From dump file:

FAULTING_IP: 
nt!ViCtxCheckAndReleaseXSaveData+5a
817125a4 cd2c            int     2Ch

I think it is a differnt issues. Are you running the VMs with -xsafe for CPU options? That should be done for the old 32bit Windows.
Comment 8 xiagao 2020-08-26 10:07:01 UTC 
(In reply to Yan Vugenfirer from comment #7)
> (In reply to xiagao from comment #5)
> > I run win832 in hck, hit BSOD.
> > Could you help check if they are the same issue?
> > 
> > DUMP file is in below link.
> > http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/
> > netkvm-dump/dump-win832.tar.gz
> > 
> > It happened during running "Configure Driver Verifier for Device Test"
> 
> From dump file:
> 
> FAULTING_IP: 
> nt!ViCtxCheckAndReleaseXSaveData+5a
> 817125a4 cd2c            int     2Ch
> 
> I think it is a differnt issues. Are you running the VMs with -xsafe for CPU
> options? That should be done for the old 32bit Windows.

No, I don't use -xsave cpu flag.
-cpu EPYC,hv_stimer,hv_synic,hv_time,hv_vpindex,hv_relaxed,hv_spinlocks=0xfff,hv_vapic,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi \
Comment 9 xiagao 2020-08-27 09:27:49 UTC 
(In reply to Yan Vugenfirer from comment #7)
> (In reply to xiagao from comment #5)
> > I run win832 in hck, hit BSOD.
> > Could you help check if they are the same issue?
> > 
> > DUMP file is in below link.
> > http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/
> > netkvm-dump/dump-win832.tar.gz
> > 
> > It happened during running "Configure Driver Verifier for Device Test"
> 
> From dump file:
> 
> FAULTING_IP: 
> nt!ViCtxCheckAndReleaseXSaveData+5a
> 817125a4 cd2c            int     2Ch
> 
> I think it is a differnt issues. Are you running the VMs with -xsafe for CPU
> options? That should be done for the old 32bit Windows.

Thanks. 
Didn't hit BSOD with "-xsave" for win8-32/64.
Comment 10 xiagao 2020-09-01 07:57:34 UTC 
(In reply to xiagao from comment #9)
> (In reply to Yan Vugenfirer from comment #7)
> > (In reply to xiagao from comment #5)
> > > I run win832 in hck, hit BSOD.
> > > Could you help check if they are the same issue?
> > > 
> > > DUMP file is in below link.
> > > http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/
> > > netkvm-dump/dump-win832.tar.gz
> > > 
> > > It happened during running "Configure Driver Verifier for Device Test"
> > 
> > From dump file:
> > 
> > FAULTING_IP: 
> > nt!ViCtxCheckAndReleaseXSaveData+5a
> > 817125a4 cd2c            int     2Ch
> > 
> > I think it is a differnt issues. Are you running the VMs with -xsafe for CPU
> > options? That should be done for the old 32bit Windows.
> 
> Thanks. 
> Didn't hit BSOD with "-xsave" for win8-32/64.

Yan, you said "That should be done for the old 32bit Windows", but I also hit BSOD on win8-64, and pass with "-xsave".
So does it also work for 64bit Windows?
Comment 11 Yvugenfi@redhat.com 2020-09-01 08:01:40 UTC 
(In reply to xiagao from comment #10)
> (In reply to xiagao from comment #9)
> > (In reply to Yan Vugenfirer from comment #7)
> > > (In reply to xiagao from comment #5)
> > > > I run win832 in hck, hit BSOD.
> > > > Could you help check if they are the same issue?
> > > > 
> > > > DUMP file is in below link.
> > > > http://fileshare.englab.nay.redhat.com/pub/section2/coredump/virtio-win/
> > > > netkvm-dump/dump-win832.tar.gz
> > > > 
> > > > It happened during running "Configure Driver Verifier for Device Test"
> > > 
> > > From dump file:
> > > 
> > > FAULTING_IP: 
> > > nt!ViCtxCheckAndReleaseXSaveData+5a
> > > 817125a4 cd2c            int     2Ch
> > > 
> > > I think it is a differnt issues. Are you running the VMs with -xsafe for CPU
> > > options? That should be done for the old 32bit Windows.
> > 
> > Thanks. 
> > Didn't hit BSOD with "-xsave" for win8-32/64.
> 
> Yan, you said "That should be done for the old 32bit Windows", but I also
> hit BSOD on win8-64, and pass with "-xsave".
> So does it also work for 64bit Windows?

Can you please post the  crash dump for 64bit as well. The previous issue happened only on 32bit systems, as far as I remeber.

Thanks.
Comment 12 Yvugenfi@redhat.com 2020-09-01 08:40:12 UTC 
Please open addtional BZ for the -xsave crashes.

Thanks.
Comment 13 ybendito 2020-09-01 08:45:17 UTC 
Note that the BZ describes problem with 64-bit (an probably ANY) Windows
Fix for the problem described in this BZ is merged to upstream.

https://github.com/virtio-win/kvm-guest-drivers-windows/commit/95918ea892b41afea1b0d3663995eda17e387232
Comment 14 xiagao 2020-09-01 10:17:33 UTC 
(In reply to Yan Vugenfirer from comment #12)
> Please open addtional BZ for the -xsave crashes.
> 
> Thanks.

(In reply to Yan Vugenfirer from comment #12)
> Please open addtional BZ for the -xsave crashes.
> 
> Thanks.

Sorry, I can't reproduce Bsod with win8-64,maybe I make some mistake before.

BTW, you mean open additional BZ for the -xsave crashes for win832, right?
Comment 15 Yvugenfi@redhat.com 2020-09-01 12:11:44 UTC 
(In reply to xiagao from comment #14)
> (In reply to Yan Vugenfirer from comment #12)
> > Please open addtional BZ for the -xsave crashes.
> > 
> > Thanks.
> 
> (In reply to Yan Vugenfirer from comment #12)
> > Please open addtional BZ for the -xsave crashes.
> > 
> > Thanks.
> 
> Sorry, I can't reproduce Bsod with win8-64,maybe I make some mistake before.
> 
> BTW, you mean open additional BZ for the -xsave crashes for win832, right?

If you are not reproducing the issue on 64bit OS, then no need to open new BZ.

Thanks,
Yan.
Comment 16 xiagao 2020-09-07 10:22:34 UTC 
(In reply to xiagao from comment #4)
> I run win2019 guest in hlk-1809 with 189 driver for 16 times, all pass,
> didn't reproduce this issue.

Hello Yuri,
I didn't reproduce this issue in my test env according to comment 0, do you have any special steps/config to reproduce it?

Thanks,
Xiaoling
Comment 17 ybendito 2020-09-08 07:26:31 UTC 
(In reply to xiagao from comment #16)
> (In reply to xiagao from comment #4)
> > I run win2019 guest in hlk-1809 with 189 driver for 16 times, all pass,
> > didn't reproduce this issue.
> 
> Hello Yuri,
> I didn't reproduce this issue in my test env according to comment 0, do you
> have any special steps/config to reproduce it?

Unfortunately we do not have a clear way to reproduce the problem.
It happened at least once (we know that from the test status and dump file left after the test).

> 
> Thanks,
> Xiaoling
Comment 24 Vadim Rozenfeld 2020-11-12 05:04:55 UTC 
should be fixed in build 190
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1383208
Comment 25 Peixiu Hou 2020-11-17 03:53:57 UTC 
Hi Yuri,

I tried to reproduce this bug with 189 build on win2019\win10-64-2004\win10-64-1809 vms. 
Enable S4, and start VM automatically after it gets to S4 (which means qemu exit), run the job "DF - Sleep with IO During (Reliability)" 10 times on each vm, did not reproduce this bug.

Run this job with virtio-win-prewhql-190 build on win2019\win10-64-2004\win10-64-1809 vms, 10 times on each vm, all passed.

I'm not sure if it can be set to verified base upper results?

Test cmds:
-netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device e1000e,bus=root1.0,netdev=hostnet0,id=net0,mac=00:52:4a:03:90:8a \
-netdev tap,script=/etc/qemu-ifup-private,downscript=no,id=hostnet1,vhost=on,queues=8 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=00:52:71:74:52:19,bus=root2.0,mq=on,vectors=18 \
-monitor stdio -spice id=on,disable-ticketing,port=5912 -vga qxl -global ICH9-LPC.disable_s4=0

Used versions:
For win2019 && win10-64-1809:
kernel-4.18.0-193.el8.x86_64
qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
seabios-bin-1.13.0-1.module+el8.2.0+5520+4e5817f3.noarch
virtio-win-prewhql-190

For win10-64-2004:
kernel-4.18.0-240.4.1.el8_3.x86_64
qemu-kvm-5.1.0-13.module+el8.3.0+8424+e82f331d.x86_64
seabios-bin-1.14.0-1.module+el8.3.0+7638+07cf13d2.noarch
virtio-win-prewhql-190

Btw, could help answer comment#23 as well? 

Thanks a lot~
Peixiu
Comment 28 ybendito 2020-11-19 13:32:25 UTC 
(In reply to Peixiu Hou from comment #25)
> Hi Yuri,
> 
> I tried to reproduce this bug with 189 build on
> win2019\win10-64-2004\win10-64-1809 vms. 
> Enable S4, and start VM automatically after it gets to S4 (which means qemu
> exit), run the job "DF - Sleep with IO During (Reliability)" 10 times on
> each vm, did not reproduce this bug.
> 
> Run this job with virtio-win-prewhql-190 build on
> win2019\win10-64-2004\win10-64-1809 vms, 10 times on each vm, all passed.
> 
> I'm not sure if it can be set to verified base upper results?
> 
> Test cmds:
> -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0 -device
> e1000e,bus=root1.0,netdev=hostnet0,id=net0,mac=00:52:4a:03:90:8a \
> -netdev
> tap,script=/etc/qemu-ifup-private,downscript=no,id=hostnet1,vhost=on,
> queues=8 -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=00:52:71:74:52:19,bus=root2.0,
> mq=on,vectors=18 \
> -monitor stdio -spice id=on,disable-ticketing,port=5912 -vga qxl -global
> ICH9-LPC.disable_s4=0
> 
> Used versions:
> For win2019 && win10-64-1809:
> kernel-4.18.0-193.el8.x86_64
> qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
> seabios-bin-1.13.0-1.module+el8.2.0+5520+4e5817f3.noarch
> virtio-win-prewhql-190
> 
> For win10-64-2004:
> kernel-4.18.0-240.4.1.el8_3.x86_64
> qemu-kvm-5.1.0-13.module+el8.3.0+8424+e82f331d.x86_64
> seabios-bin-1.14.0-1.module+el8.3.0+7638+07cf13d2.noarch
> virtio-win-prewhql-190
> 

Unfortunately I can't provide any guidelines how to reproduce this bug.
It happened once in our HCK-CI environment which runs regular HCK under automation.
From dump analysis and code review we found the problem and fixed it.
From then the problem was never seen.
Probably you can change it to verified based on your results and my comment.
Comment 29 Peixiu Hou 2020-11-23 01:31:40 UTC 
Based on comment#25 and comment#28, set this bug to verified.

Thanks~
Peixiu
Comment 32 errata-xmlrpc 2021-02-16 14:24:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virtio-win bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:0535