Description of problem: recently I am focusing on hyper-v testing .For win10-32(q35) guest , It may fail in auto with two following scenarios: 1. After add flag 'hv_vapic', improvement not above 5%. (from the log it is 1%--4% sometimes) 2. No improvement at all and the performance seems worse after add flag 'hv_vapic' . Version-Release number of selected component (if applicable): qemu-kvm-6.0.0-27.module+el8.5.0+12121+c40c8708.x86_64 kernel-4.18.0-330.el8.x86_64 seabios-bin-1.14.0-1.module+el8.4.0+8855+a9e237a9.noarch How reproducible: not 100% Steps to Reproduce: 1. Boot the windows guest with all the flags MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35,memory-backend=mem-machine_mem \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 14336 \ -object memory-backend-ram,size=14336M,id=mem-machine_mem \ -smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2 \ -cpu 'Skylake-Server',hv_crash,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \ -chardev socket,id=qmp_id_qmpmonitor1,wait=off,path=/tmp/avocado_xqla13n_/monitor-qmpmonitor1-20210816-232807-w8StSmYx,server=on \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,wait=off,path=/tmp/avocado_xqla13n_/monitor-catch_monitor-20210816-232807-w8StSmYx,server=on \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idzWI6Vu \ -chardev socket,id=chardev_serial0,wait=off,path=/tmp/avocado_xqla13n_/serial-serial0-20210816-232807-w8StSmYx,server=on \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20210816-232807-w8StSmYx,path=/tmp/avocado_xqla13n_/seabios-20210816-232807-w8StSmYx,server=on,wait=off \ -device isa-debugcon,chardev=seabioslog_id_20210816-232807-w8StSmYx,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/win10-32-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \ -blockdev node-name=file_disk1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/tmp/avocado_xqla13n_/tmpfs/data.raw,cache.direct=off,cache.no-flush=off \ -blockdev node-name=drive_disk1,driver=raw,read-only=off,cache.direct=off,cache.no-flush=off,file=file_disk1 \ -device ide-hd,id=disk1,drive=drive_disk1,write-cache=off,bus=ide.0,unit=0 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:92:d6:53:b5:57,id=idV8FDSJ,netdev=idYxrySV,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idYxrySV,vhost=on,vhostfd=20,fd=16 \ -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \ -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \ -vnc :0 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 2. Partition hard drive, create an NTFS partition 3. disable write caching on the hard drive in Windows or reformat the disk at the second time (device manager -> Disk drives -> Pick the second QEMU HARDDISK -> Policies) 4. Install FIO (https://bsdio.com/fio/) 5. send command "C:\Program Files\fio\fio\fio.exe" --name=fio-rand-RW --filename=fio-rand-RW --directory=E\:\ --rw=randwrite --bs=512B --direct=1 --numjobs=1 --time_based=1 --runtime=300 --size=1G --iodepth=1 7. check the r/w result 8. Reboot the guest without 'hv_vapic', let it calm down and run the same job 9. Check the improvement of hv_vapic Actual results: Improvement not above 5% with hv_vapic Expected results: Improvement should above 5% with hv_vapic Additional info: Also hit this issue with win2019 and win2016 recently
Assigned to Amnon for initial triage per bz process and age of bug created or assigned to virt-maint without triage.
Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
hit the same issue on win11(ovmf) Packages: qemu-kvm-7.0.0-1.el9.x86_64 kernel-5.14.0-78.el9.x86_64 seabios-bin-1.15.0-1.el9.noarch RHEL-9.1.0-20220419.0
hit the same issue on win11(ovmf) Packages: qemu-kvm-7.2.0-1.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20220826gitba0e0e4c6a-2.el9.noarch kernel-5.14.0-212.el9.x86_64 RHEL-9.2.0-20221219.0
Reproduced this issue on RHEL9.2 host, the performance of the storage has no obviously improvement. I pasted the fio results as follows: 1) With hv_vapic flag: ============================================================================================================== C:\Program Files\fio>fio.exe --name=fio-rand-RW --directory=D\:\ --rw=randwrite --bs=512B --direct=1 --numjobs=1 --time_based=1 --runtime=300 --size=1G --iodepth=1 fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning. fio-rand-RW: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=windowsaio, iodepth=1 fio-3.18 Starting 1 thread fio-rand-RW: Laying out IO file (1 file / 1024MiB) Jobs: 1 (f=0): [f(1)][100.0%][w=4702KiB/s][w=9405 IOPS][eta 00m:00s] fio-rand-RW: (groupid=0, jobs=1): err= 0: pid=1044: Wed Feb 15 02:52:53 2023 write: IOPS=10.9k, BW=5455KiB/s (5586kB/s)(1598MiB/300001msec) slat (usec): min=18, max=1117.7k, avg=20.84, stdev=629.08 clat (nsec): min=300, max=3002.1k, avg=69764.46, stdev=6455.55 lat (usec): min=70, max=1117.7k, avg=90.60, stdev=629.06 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 68], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 71], 80.00th=[ 72], 90.00th=[ 73], 95.00th=[ 74], | 99.00th=[ 80], 99.50th=[ 85], 99.90th=[ 94], 99.95th=[ 118], | 99.99th=[ 347] bw ( KiB/s): min= 0, max= 5569, per=100.00%, avg=5461.72, stdev=362.18, samples=556 iops : min= 1, max=11139, avg=10923.86, stdev=724.33, samples=556 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.02%, 50=0.03% lat (usec) : 100=99.84%, 250=0.05%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=2.67%, sys=23.33%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3273198,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=5455KiB/s (5586kB/s), 5455KiB/s-5455KiB/s (5586kB/s-5586kB/s), io=1598MiB (1676MB), run=300001-300001msec 2) Without hv_vapic flag: ============================================================================================================= C:\Program Files\fio>fio.exe --name=fio-rand-RW --directory=D\:\ --rw=randwrite --bs=512B --direct=1 --numjobs=1 --time_based=1 --runtime=300 --size=1G --iodepth=1 fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning. fio-rand-RW: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=windowsaio, iodepth=1 fio-3.18 Starting 1 thread Jobs: 1 (f=0): [f(1)][100.0%][w=5217KiB/s][w=10.4k IOPS][eta 00m:00s] fio-rand-RW: (groupid=0, jobs=1): err= 0: pid=2876: Wed Feb 15 03:18:26 2023 write: IOPS=11.0k, BW=5512KiB/s (5645kB/s)(1615MiB/300001msec) slat (usec): min=17, max=5380, avg=20.37, stdev= 6.44 clat (nsec): min=300, max=18956k, avg=69288.91, stdev=15765.84 lat (usec): min=63, max=18997, avg=89.66, stdev=17.13 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 68], 60.00th=[ 70], | 70.00th=[ 72], 80.00th=[ 76], 90.00th=[ 79], 95.00th=[ 82], | 99.00th=[ 90], 99.50th=[ 94], 99.90th=[ 110], 99.95th=[ 155], | 99.99th=[ 400] bw ( KiB/s): min= 4875, max= 6030, per=100.00%, avg=5519.22, stdev=224.52, samples=558 iops : min= 9751, max=12060, avg=11038.93, stdev=449.02, samples=558 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.04% lat (usec) : 100=99.73%, 250=0.15%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=2.67%, sys=22.67%, ctx=0, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3307383,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=5512KiB/s (5645kB/s), 5512KiB/s-5512KiB/s (5645kB/s-5645kB/s), io=1615MiB (1693MB), run=300001-300001msec Used Versions: OS: Windows 2022-64 kernel-5.14.0-249.el9.x86_64 qemu-kvm-7.2.0-5.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch virtio-win-1.9.31-0.el9_1.iso And BTW: Hi Vitaly, Could you help to expand the Stale Date? The date is 2023-02-17 now, It's going to be closed automatically~ Thanks～ Peixiu
Added +6 month to the Stale Date, hope we'll get to investigate this for 9.3. This probably needs to be checked in conjunction with 'hv_vapic'/'hv_apicv' as PV EOI feature is not used with APICv/AVIC.
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Hi Vitaly, This issue was auto closed, As comment#11, we also can reproduce it. Do we have fix plan for that? if yes, I'll reopen it again. Thanks~ Peixiu