Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2173423

Summary: [Win10.x86_64] Win10 64 bit guest gets stuck while booting sometimes.
Product: Red Hat Enterprise Linux 9 Reporter: liunana <nanliu>
Component: virtio-winAssignee: Yvugenfi <yvugenfi>
virtio-win sub component: others QA Contact: Yiqian Wei <yiwei>
Status: CLOSED MIGRATED Docs Contact:
Severity: high    
Priority: unspecified CC: chayang, coli, jinzhao, qizhu, virt-maint
Version: 9.2Keywords: MigratedToJIRA
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-15 06:11:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liunana 2023-02-26 15:42:59 UTC
Description of problem:
Win10 64 bit guest gets stuck while booting sometimes.


Version-Release number of selected component (if applicable):
Host:
    5.14.0-277.el9.x86_64
    qemu-kvm-7.2.0-10.el9.x86_64
    edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
    Model name:            Intel(R) Xeon(R) Gold 6414TDX
Guest: ovmf windows 10.x86_64


How reproducible: 1/6


Steps to Reproduce:
1. Boot a win 10 vm, full qemu command [1]
2. repeat step 1
3.

Actual results:
Win10 guest gets stuck sometims. And can recover it by system_reset

Expected results:
Win10 won't get stuck while booting up.

Additional info:
[1]
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
    -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_win10-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
    -m 62464 \
    -object '{"size": 65498251264, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
    -cpu 'Icelake-Server-noTSX',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,wait=off,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_4wkp6ukn/monitor-qmpmonitor1-20230226-101530-guM6M4yY,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,id=qmp_id_catch_monitor,path=/var/tmp/avocado_4wkp6ukn/monitor-catch_monitor-20230226-101530-guM6M4yY,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device '{"ioport": 1285, "driver": "pvpanic", "id": "idE43ELo"}' \
    -chardev socket,wait=off,id=chardev_serial0,path=/var/tmp/avocado_4wkp6ukn/serial-serial0-20230226-101530-guM6M4yY,server=on \
    -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
    -chardev socket,id=seabioslog_id_20230226-101530-guM6M4yY,path=/var/tmp/avocado_4wkp6ukn/seabios-20230226-101530-guM6M4yY,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20230226-101530-guM6M4yY,iobase=0x402 \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"driver": "virtio-net-pci", "mac": "9a:d3:b3:70:4e:9d", "id": "idvhykOE", "netdev": "idLXXOSd", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
    -netdev tap,id=idLXXOSd,vhost=on,vhostfd=16,fd=12 \
    -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
    -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/home/kar/workspace/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
    -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
    -device '{"id": "tpm-crb_vtpm_avocado-vt-vm1_tpm0", "tpmdev": "emulator_vtpm_avocado-vt-vm1_tpm0", "driver": "tpm-crb"}' \
    -enable-kvm \

Comment 1 liunana 2023-02-26 15:52:23 UTC
Also can reproduce this issue without hyper-v flags with '-cpu 'Icelake-Server-noTSX',+kvm_pv_unhalt'.

Comment 2 liunana 2023-02-27 02:11:58 UTC
Can't reproduce this issue with Win2022,Win2016,Win2019 by over 100 times with hyper-v flags.

Comment 3 liunana 2023-02-27 03:54:29 UTC
Don't know if this matters, but I didn't reproduce this issue by over 100 times with hyper-v flags on AMD Genoa host.

Host: 5.14.0-277.el9.x86_64
      qemu-kvm-7.2.0-10.el9.x86_64
      Model name:            AMD EPYC 9754 128-Core Processor
      edk2-ovmf-20221207gitfff6d81270b5-7.el9.noarch
      libvirt-devel-9.0.0-6.el9.x86_64


qemu commandline:
     -S  \
     -name 'avocado-vt-vm1'  \
     -sandbox on  \
     -blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
     -blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/home/kar/workspace/root/avocado/data/avocado-vt/avocado-vt-vm1_win10-64-virtio-scsi_qcow2_filesystem_VARS.fd", "auto-read-only": true, "discard": "unmap"}' \
     -blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
     -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
     -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
     -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
     -nodefaults \
     -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
     -m 61440 \
     -object '{"size": 64424509440, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
     -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
     -cpu 'EPYC-Milan',x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,avx512f=on,avx512dq=on,avx512ifma=on,avx512cd=on,avx512bw=on,avx512vl=on,avx512vbmi=on,avx512vbmi2=on,gfni=on,vaes=on,vpclmulqdq=on,avx512vnni=on,avx512bitalg=on,avx512-vpopcntdq=on,la57=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,avx512-bf16=on,cmp-legacy=on,virt-ssbd=on,lbrv=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/monitor-qmpmonitor1-20230226-105448-g4QDCe6k,wait=off,id=qmp_id_qmpmonitor1  \
     -mon chardev=qmp_id_qmpmonitor1,mode=control \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/monitor-catch_monitor-20230226-105448-g4QDCe6k,wait=off,id=qmp_id_catch_monitor  \
     -mon chardev=qmp_id_catch_monitor,mode=control \
     -device '{"ioport": 1285, "driver": "pvpanic", "id": "idkABW80"}' \
     -chardev socket,server=on,path=/var/tmp/avocado_yrugfshf/serial-serial0-20230226-105448-g4QDCe6k,wait=off,id=chardev_serial0 \
     -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
     -chardev socket,id=seabioslog_id_20230226-105448-g4QDCe6k,path=/var/tmp/avocado_yrugfshf/seabios-20230226-105448-g4QDCe6k,server=on,wait=off \
     -device isa-debugcon,chardev=seabioslog_id_20230226-105448-g4QDCe6k,iobase=0x402 \
     -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
     -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
     -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
     -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
     -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
     -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
     -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
     -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
     -device '{"driver": "virtio-net-pci", "mac": "9a:ea:4a:d8:27:8c", "id": "idksPx4S", "netdev": "idse84ks", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
     -netdev tap,id=idse84ks,vhost=on,vhostfd=16,fd=12 \
     -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
     -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
     -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
     -vnc :0  \
     -rtc base=localtime,clock=host,driftfix=slew  \
     -boot menu=off,order=cdn,once=c,strict=off \
     -chardev socket,id=char_vtpm_avocado-vt-vm1_tpm0,path=/home/kar/workspace/root/avocado/data/avocado-vt/swtpm/avocado-vt-vm1_tpm0_swtpm.sock \
     -tpmdev emulator,chardev=char_vtpm_avocado-vt-vm1_tpm0,id=emulator_vtpm_avocado-vt-vm1_tpm0 \
     -device '{"id": "tpm-crb_vtpm_avocado-vt-vm1_tpm0", "tpmdev": "emulator_vtpm_avocado-vt-vm1_tpm0", "driver": "tpm-crb"}' \
     -enable-kvm \

Comment 4 liunana 2023-02-27 07:38:25 UTC
Can reproduce this issue with seabios win10 64 bit vm by rebooting.
How reproducible: 1/6

qemu command line:
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
    -m 62464 \
    -object '{"size": 65498251264, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
    -cpu 'Icelake-Server-noTSX',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/monitor-qmpmonitor1-20230226-231825-nfA5nWPw,wait=off,id=qmp_id_qmpmonitor1,server=on  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/monitor-catch_monitor-20230226-231825-nfA5nWPw,wait=off,id=qmp_id_catch_monitor,server=on  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device '{"ioport": 1285, "driver": "pvpanic", "id": "idMRat87"}' \
    -chardev socket,path=/var/tmp/avocado_9ezejv95/serial-serial0-20230226-231825-nfA5nWPw,wait=off,id=chardev_serial0,server=on \
    -device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
    -chardev socket,id=seabioslog_id_20230226-231825-nfA5nWPw,path=/var/tmp/avocado_9ezejv95/seabios-20230226-231825-nfA5nWPw,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20230226-231825-nfA5nWPw,iobase=0x402 \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-2", "addr": "0x0"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/win10-64-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"driver": "virtio-net-pci", "mac": "9a:6b:88:41:b3:c7", "id": "idP9RGfL", "netdev": "idsxaTpc", "bus": "pcie-root-port-3", "addr": "0x0"}'  \
    -netdev tap,id=idsxaTpc,vhost=on,vhostfd=16,fd=12 \
    -blockdev '{"node-name": "file_cd1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/iso/windows/winutils.iso", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_cd1", "driver": "raw", "read-only": true, "cache": {"direct": true, "no-flush": false}, "file": "file_cd1"}' \
    -device '{"driver": "scsi-cd", "id": "cd1", "drive": "drive_cd1", "write-cache": "on"}'  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \

Comment 5 Yvugenfi@redhat.com 2023-05-01 08:12:19 UTC
Hello, can you please generate a crash dump on a guest?
1. Please configure the guest to save the kernel crash dump
2. Send "nmi" command through QEMU monitor

Comment 6 liunana 2023-07-06 11:50:45 UTC
(In reply to Yvugenfi from comment #5)
> Hello, can you please generate a crash dump on a guest?
> 1. Please configure the guest to save the kernel crash dump
> 2. Send "nmi" command through QEMU monitor

Hi Yan,

I'm sorry for the later reply, I can't reproduce it recently, and I'm keeping trying it now.
I will update crash dump log once I reproduce this issue.
And if I still can't reproduce it with many tries, we can consider closing it.

Thanks.


Best regards
Nana

Comment 8 RHEL Program Management 2023-08-14 07:27:43 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.