Bug 2173423

Summary: [Win10.x86_64] Win10 64 bit guest gets stuck while booting sometimes.
Product: Red Hat Enterprise Linux 9 Reporter: liunana <nanliu>
Component: virtio-winAssignee: Yvugenfi <yvugenfi>
virtio-win sub component: others QA Contact: Yiqian Wei <yiwei>
Status: CLOSED MIGRATED Docs Contact:
Severity: high    
Priority: unspecified CC: chayang, coli, jinzhao, qizhu, virt-maint
Version: 9.2Keywords: MigratedToJIRA
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-15 06:11:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liunana 2023-02-26 15:42:59 UTC
Description of problem:
Win10 64 bit guest gets stuck while booting sometimes.


Version-Release number of selected component (if applicable):
Host:
    5.14.0-277.el9.x86_64
    qemu-kvm-7.2.0-10.el9.x86_64
    edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
    Model name:            Intel(R) Xeon(R) Gold 6414TDX
Guest: ovmf windows 10.x86_64


How reproducible: 1/6


Steps to Reproduce:
1. Boot a win 10 vm, full qemu command [1]
2. repeat step 1
3.

Actual results:
Win10 guest gets stuck sometims. And can recover it by system_reset

Expected results:
Win10 won't get stuck while booting up.

Additional info:
[1]
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
    -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_win10-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
    -m 62464 \
    -object '{"size": 65498251264, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
    -cpu 'Icelake-Server-noTSX',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,wait=off,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_4wkp6ukn/monitor-qmpmonitor1-20230226-101530-guM6M4yY,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,id=qmp_id_catch_monitor,path=/var/tmp/avocado_4wkp6ukn/monitor-catch_monitor-20230226-101530-guM6M4yY,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device '{"ioport": 1285, "driver": "pvpanic", "id": "idE43ELo"}' \
    -chardev socket,wait=off,id=chardev_serial0,path=/var/tmp/avocado_4wkp6ukn/serial-serial0-20230226-101530-guM6M4yY,server=on \
    -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
    -chardev socket,id=seabioslog_id_20230226-101530-guM6M4yY,path=/var/tmp/avocado_4wkp6ukn/seabios-20230226-101530-guM6M4yY,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20230226-101530-guM6M4yY,iobase=0x402 \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"driver": "virtio-net-pci", "mac": "9a:d3:b3:70:4e:9d", "id": "idvhykOE", "netdev": "idLXXOSd", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
    -netdev tap,id=idLXXOSd,vhost=on,vhostfd=16,fd=12 \
    -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
    -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/home/kar/workspace/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
    -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
    -device '{"id": "tpm-crb_vtpm_avocado-vt-vm1_tpm0", "tpmdev": "emulator_vtpm_avocado-vt-vm1_tpm0", "driver": "tpm-crb"}' \
    -enable-kvm \

Comment 1 liunana 2023-02-26 15:52:23 UTC
Also can reproduce this issue without hyper-v flags with '-cpu 'Icelake-Server-noTSX',+kvm_pv_unhalt'.

Comment 2 liunana 2023-02-27 02:11:58 UTC
Can't reproduce this issue with Win2022,Win2016,Win2019 by over 100 times with hyper-v flags.

Comment 3 liunana 2023-02-27 03:54:29 UTC
Don't know if this matters, but I didn't reproduce this issue by over 100 times with hyper-v flags on AMD Genoa host.

Host: 5.14.0-277.el9.x86_64
      qemu-kvm-7.2.0-10.el9.x86_64
      Model name:            AMD EPYC 9754 128-Core Processor
      edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
      libvirt-devel-9.0.0-6.el9.x86_64


qemu commandline:
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_win10-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 61440 \
     -object '{"size": 64424509440, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
     -cpu 'EPYC-Milan',x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,avx512f=on,avx512dq=on,avx512ifma=on,avx512cd=on,avx512bw=on,avx512vl=on,avx512vbmi=on,avx512vbmi2=on,gfni=on,vaes=on,vpclmulqdq=on,avx512vnni=on,avx512bitalg=on,avx512-vpopcntdq=on,la57=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,avx512-bf16=on,cmp-legacy=on,virt-ssbd=on,lbrv=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/monitor-qmpmonitor1-20230226-105448-g4QDCe6k,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/monitor-catch_monitor-20230226-105448-g4QDCe6k,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idkABW80"}' \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/serial-serial0-20230226-105448-g4QDCe6k,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230226-105448-g4QDCe6k,path=/var/tmp/avocado_yrugfshf/seabios-20230226-105448-g4QDCe6k,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230226-105448-g4QDCe6k,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:ea:4a:d8:27:8c", "id": "idksPx4S", "netdev": "idse84ks", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=idse84ks,vhost=on,vhostfd=16,fd=12 \
     -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
     -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
     -vnc :0  \
     -rtc base=localtime,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/home/kar/workspace/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
     -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
     -device '{"id": "tpm-crb_vtpm_avocado-vt-vm1_tpm0", "tpmdev": "emulator_vtpm_avocado-vt-vm1_tpm0", "driver": "tpm-crb"}' \
     -enable-kvm \

Comment 4 liunana 2023-02-27 07:38:25 UTC
Can reproduce this issue with seabios win10 64 bit vm by rebooting.
How reproducible: 1/6

qemu command line:
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
    -m 62464 \
    -object '{"size": 65498251264, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
    -cpu 'Icelake-Server-noTSX',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/monitor-qmpmonitor1-20230226-231825-nfA5nWPw,wait=off,id=qmp_id_qmpmonitor1,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/monitor-catch_monitor-20230226-231825-nfA5nWPw,wait=off,id=qmp_id_catch_monitor,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device '{"ioport": 1285, "driver": "pvpanic", "id": "idMRat87"}' \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/serial-serial0-20230226-231825-nfA5nWPw,wait=off,id=chardev_serial0,server=on \
    -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
    -chardev socket,id=seabioslog_id_20230226-231825-nfA5nWPw,path=/var/tmp/avocado_9ezejv95/seabios-20230226-231825-nfA5nWPw,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20230226-231825-nfA5nWPw,iobase=0x402 \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"driver": "virtio-net-pci", "mac": "9a:6b:88:41:b3:c7", "id": "idP9RGfL", "netdev": "idsxaTpc", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
    -netdev tap,id=idsxaTpc,vhost=on,vhostfd=16,fd=12 \
    -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
    -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \

Comment 5 Yvugenfi@redhat.com 2023-05-01 08:12:19 UTC
Hello, can you please generate a crash dump on a guest?
1. Please configure the guest to save the kernel crash dump
2. Send "nmi" command through QEMU monitor

Comment 6 liunana 2023-07-06 11:50:45 UTC
(In reply to Yvugenfi from comment #5)
> Hello, can you please generate a crash dump on a guest?
> 1. Please configure the guest to save the kernel crash dump
> 2. Send "nmi" command through QEMU monitor

Hi Yan,

I'm sorry for the later reply, I can't reproduce it recently, and I'm keeping trying it now.
I will update crash dump log once I reproduce this issue.
And if I still can't reproduce it with many tries, we can consider closing it.

Thanks.


Best regards
Nana

Comment 8 RHEL Program Management 2023-08-14 07:27:43 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.