Bug 1667330

Created attachment 1521428 [details]
long time in shutting down page

Created attachment 1521430 [details]
Download the windows system event log from win2012-r2 guest

*** Bug 1683599 has been marked as a duplicate of this bug. ***

Can we check if the problem is reproducible with the latest sdrivers from build 170?
http://download.eng.bos.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/170/win/virtio-win-prewhql-0.1.zip

Thanks,
Vadim.

(In reply to Vadim Rozenfeld from comment #7)
> Can we check if the problem is reproducible with the latest sdrivers from
> build 170?
> http://download.eng.bos.redhat.com/brewroot/packages/virtio-win-prewhql/0.1/
> 170/win/virtio-win-prewhql-0.1.zip
> 
> Thanks,
> Vadim.

Hi vadim,

Tried testwith virtio-win-prewhql-170 on win2012-r2 geust, it also can be reproduced, I tried run 100 times with automation, hit this issue 40 times.

Best Regards~
Peixiu

hit a similar issue for case 'migration_with_block.with_dataplane_on2off.send_shell.shutdown_vm' in auto test,failed to shutdown vm,just track it here.

Host:
qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
kernel-4.18.0-193.el8.x86_649bf
seabios-1.13.0-1.module+el8.2.0+5520+4e5817f3.x86_64

Guest:
windows 2016(q35) with virtio-win-prewhql-181.iso

(In reply to menli from comment #10)
> hit a similar issue for case
> 'migration_with_block.with_dataplane_on2off.send_shell.shutdown_vm' in auto
> test,failed to shutdown vm,just track it here.
> 
> Host:
> qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
> kernel-4.18.0-193.el8.x86_649bf
> seabios-1.13.0-1.module+el8.2.0+5520+4e5817f3.x86_64
> 
> Guest:
> windows 2016(q35) with virtio-win-prewhql-181.iso

1. ceate 40G data disk
qemu-img create -f qcow2 /home/kvm_autotest_root/images/storage0.qcow2 40G

2. Boot guest with dataplane on src host

MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 14336  \
    -smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/monitor-qmpmonitor1-20200429-015214-HL3oizxq,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/monitor-catch_monitor-20200429-015214-HL3oizxq,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id3UJFmZ \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/serial-serial0-20200429-015214-HL3oizxq,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200429-015214-HL3oizxq,path=/var/tmp/avocado_3s1051dg/seabios-20200429-015214-HL3oizxq,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200429-015214-HL3oizxq,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/win2016-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,iothread=iothread0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/storage0.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-blk-pci,id=stg0,drive=drive_stg0,bootindex=1,write-cache=on,iothread=iothread1,bus=pcie-root-port-3,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:49:63:b8:8b:03,id=id4BeT93,netdev=id3m5Beo,bus=pcie-root-port-4,addr=0x0  \
    -netdev tap,id=id3m5Beo,vhost=on,vhostfd=21,fd=15 \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=2,write-cache=on,bus=ide.0,unit=0  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=6

3. Boot guest without dataplane on dst host.

MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 14336  \
    -smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/monitor-qmpmonitor1-20200429-015247-jtpIzwSr,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/monitor-catch_monitor-20200429-015247-jtpIzwSr,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idZ7GSeT \
    -chardev socket,server,nowait,path=/var/tmp/avocado_3s1051dg/serial-serial0-20200429-015247-jtpIzwSr,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200429-015247-jtpIzwSr,path=/var/tmp/avocado_3s1051dg/seabios-20200429-015247-jtpIzwSr,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200429-015247-jtpIzwSr,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/win2016-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/storage0.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-blk-pci,id=stg0,drive=drive_stg0,bootindex=1,write-cache=on,bus=pcie-root-port-3,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:49:63:b8:8b:03,id=idZTC1OV,netdev=idD0Xgvl,bus=pcie-root-port-4,addr=0x0  \
    -netdev tap,id=idD0Xgvl,vhost=on,vhostfd=42,fd=27 \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=2,write-cache=on,bus=ide.0,unit=0  \
    -vnc :1  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=6 \
    -incoming tcp:0:5846

4.do migration

5.format data disk in diskpart

6.run iozone on data disk
D:\Iozone\iozone.exe -az -b C:\E_stress_test -g 1g -y 32k -i 0 -i 1 -I -f E:\iozone_test

7.shut down guest
shutdown -t 0 -s


After step7,guest fail to shutdown.

Can you check if there is event 129 or any other storage/disk
related events in the system events log file triggered by system
during shutting down the system?

Thank you,
Vadim.

(In reply to Vadim Rozenfeld from comment #12)
> Can you check if there is event 129 or any other 
> related events in the system events log file triggered by system
> during shutting down the system?
> 
> Thank you,
> Vadim.

Hi vadim,

I check the system events log,only found error event 10016,no storage/disk related events.


Thanks

Menghuan

(In reply to menli from comment #14)
> hit the similar issue when do reboot operation after unplug a disk and hit
> BSOD.(hit this issue quite often this time)
> 
> Host:
> qemu-kvm-4.2.0-21.module+el8.2.1+6586+8b7713b9.x86_64
> kernel-4.18.0-193.el8.x86_64
> seabios-1.13.0-1.module+el8.2.0+5520+4e5817f3.x86_64
> 
> Guest:
> Win8.1-32-q35
> virtio-win-prewhql-184
> 
> The completed dump log is saved:
> 
> http://fileshare.englab.nay.redhat.com/pub/section2/coredump/bz1667330/
> Memory-win8.1.dmp
> 
> Check the system events log,found Warning Event ID 157 ，219 and error Event
> ID 1001,hope it will be useful for this issue.
> 
> 
> 
> Thanks
> Menghuan

Can you please upload the events log file as well?
Thanks,
Vadim.

Sometimes reboot failed after hotplug /unplug 
4.18.0-226.el8.x86_64
qemu-kvm-core-4.2.0-30.module+el8.3.0+7298+c26a06b8.x86_64
seabios-1.13.0-1.module+el8.3.0+6423+e4cb6418.x86_64

Test steps:
1./usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 15360  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Opteron_G5',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0xfff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -chardev socket,path=/var/tmp/avocado_pm9mpl_2/monitor-qmpmonitor1-20200722-011514-fXJXRHcC,id=qmp_id_qmpmonitor1,server,nowait  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,path=/var/tmp/avocado_pm9mpl_2/monitor-catch_monitor-20200722-011514-fXJXRHcC,id=qmp_id_catch_monitor,server,nowait  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idhEF81t \
    -chardev socket,path=/var/tmp/avocado_pm9mpl_2/serial-serial0-20200722-011514-fXJXRHcC,id=chardev_serial0,server,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200722-011514-fXJXRHcC,path=/var/tmp/avocado_pm9mpl_2/seabios-20200722-011514-fXJXRHcC,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200722-011514-fXJXRHcC,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:d9:c2:3c:da:ae,id=idr2hpK1,netdev=idoofjVJ,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idoofjVJ,vhost=on,vhostfd=21,fd=15 \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=1,write-cache=on,bus=ide.0,unit=0  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

2.hotplug disk

{"execute": "blockdev-add", "arguments": {"node-name": "file_stg0", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/storage0.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "t5gzWAv0"}
{"execute": "blockdev-add", "arguments": {"node-name": "drive_stg0", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg0"}, "id": "y5hD3thX"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "drive_stg0", "write-cache": "on", "bus": "pcie_extra_root_port_0", "addr": "0x0", "iothread": "iothread1"}, "id": "7qjgpmDO"}

3.reboot guest

4 houunplug disk
{"execute": "device_del", "arguments": {"id": "stg0"}, "id": "UlINKVie"}
{"execute": "blockdev-del", "arguments": {"node-name": "drive_stg0"}, "id": "5toNSNyg"}
{"execute": "blockdev-del", "arguments": {"node-name": "file_stg0"}, "id": "xWonev5b"}

5.reboot guest
shutdown /r /f /t 0

It will hang on step 5.

Reproduced by automation:
python ConfigTest.py --testcase=block_hotplug.block_virtio.fmt_qcow2.default.with_plug.with_reboot.one_pci.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=Win2019 --driveformat=virtio_blk --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="qemu_force_use_drive_expression = no\nimage_aio=threads\ncd_format=ide"

hit the same issue like comment 11

host:
qemu-kvm-5.1.0-15.module+el8.3.1+8772+a3fdeccd.x86_64
kernel-4.18.0-240.el8.x86_64
seabios-1.14.0-1.module+el8.3.0+7638+07cf13d2.x86_64

guest:
win8.1 32 q35, win8.1-32 pc
virtio-win-prewhql-0.1-191.iso

hit the same issue like comment 11

host:
kernel-4.18.0-278.el8.dt3.x86_64
qemu-kvm-5.2.0-4.module+el8.4.0+9676+589043b9.x86_64
virtio-win-prewhql-193
seabios-bin-1.14.0-1.module+el8.4.0+8855+a9e237a9.noarch

guest:
win10 20H2

hit the same issue on win10-64

RHEL-8.4.0-20210217.d.2
kernel-4.18.0-287.el8.dt3.x86_64
qemu-kvm-5.2.0-7.module+el8.4.0+9943+d64b3717.x86_64
seabios-1.14.0-1.module+el8.4.0+8855+a9e237a9.x86_64
virtio-win-prewhql-195


auto case: block_hotplug.block_virtio.fmt_raw.default.with_plug.with_shutdown.after_unplug.one_pci

hit the same issue on win2016（q35） like comment 11.

RHEL-8.5.0-20210609.n.3
qemu-kvm-4.2.0-51.module+el8.5.0+11141+9dff516f.x86_64
seabios-bin-1.13.0-2.module+el8.3.0+7353+9de0a3cc.noarch
kernel-4.18.0-310.el8.x86_64
virtio-win-prewhql-202


auto case: migration_with_block.default.with_dataplane_on2off.send_shell.shutdown_vm

hit the same issue on win10-32 when run virtio-win-prewhql-205 viostor testing on rhel850-av

    kernel-4.18.0-324.el8.x86_64
    qemu-kvm-6.0.0-25.module+el8.5.0+11890+8e7c3f51.x86_64
    virtio-win-prewhql-205
    seabios-bin-1.14.0-1.module+el8.4.0+8855+a9e237a9.noarch
    RHEL-8.5.0-20210727.n.0

auto case:
    block_hotplug.block_virtio.fmt_raw.with_plug.with_shutdown.after_unplug.one_pci

hit a similar issue for case 'migration_with_block.with_dataplane_on2off.send_shell.shutdown_vm' in auto test,failed to shutdown vm.

Host:
qemu-kvm-docs-6.2.0-11.el9_0.2.x86_64
kernel-5.14.0-70.7.1.el9_0.x86_64
seabios-bin-1.15.0-1.el9.noarch

Guest:
win10 21h2_32 with virtio-win-prewhql-219.iso

Hi Vadim,

Actually I didn't  hit this issue recently, but could we extend the "Stale Date" first and keep watch it for some time ?


Thanks 
Menghuan

Closing this case based on https://bugzilla.redhat.com/show_bug.cgi?id=1667330#c39.
Please feel frre to reopen it in case you have any similar issues.

Change status to close as currentrealse, since it can be reproduced till 2022-05-06, just recently, we cannot reproduce it~

Any Questions please let me known, thanks~

BR~
Peixiu