Bug 1737702

Created attachment 1604300 [details]
trace-cmd on Win10 1809 guest

trace-cmd record -e kvm on Win10 1809-32b guest

Created attachment 1604302 [details]
trace-cmd on Win10 1903 guest

trace-cmd record -e kvm on Win10 1903-32b guest

QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

I spent some time trying to reproduce the problem on 1909. 
Not sure if I can reproduce the problem with the latest Win10
version any longer. Can QE give it a try?
Thanks,
Vadim.

I can still reproduce it on win10(1909)- 64bit

If guest cpu load 0%-1%, the host cpu load for qemu-kvm 5%-10%
If guest cpu load >1% , the host cpu load for qemu-kvm >10%

qemu-kvm-4.2.0-11.module+el8.2.0+5837+4c1442ec.x86_64
kernel-4.18.0-175.el8.x86_64
seabios-1.13.0-1.module+el8.2.0+5520+4e5817f3.x86_64

Boot command:
MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -machine q35  \
    -nodefaults \
    -sandbox on  \
    -device VGA,bus=pcie.0,addr=0x1 \
    -m 6144  \
    -smp 6,maxcpus=6,cores=3,threads=1,sockets=2  \
    -cpu 'Skylake-Server',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,hv_evmcs \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_Ea97kw1,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_Ea97kw1,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id2Gz4fb \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie.0-root-port-5,addr=0x0 \
    -drive id=drive_image2,if=none,snapshot=off,aio=threads,format=raw,file=/mnt/tmpfs/data.raw \
    -device scsi-hd,id=image2,drive=drive_image2 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:57:da:4f:cb:b2,id=idotmiWW,mq=on,vectors=14,netdev=idT6Wphl,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idT6Wphl,vhost=on,queues=6 \
    -vnc :9  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=on,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:4444,server,nowait

Tried on a clean win10-64 (1909)

If the guest cpu utilization is closed to 0% for a while, the host cpu is 4%-6%.
But the service for win10 is more active than other guests. So, the result is 
not as well as others. In another words ,I cannot hold the idle status for a long
time. (sysMain service is always use the cpu)

tried with other guests (below win10), can get 4%-5%, since the guest status is stable.

Above result config:
I closed the windows defender and windows updates, disable Automatic Windows Update, 
disable NIC , unused USB ports, and CD-ROM. 


And a question for my automation scripts:

I tried with my automation scripts.The result is:

Win8-32/win8.1-64,win2016,win2019 can get 7%-9% cpu utilization in host.
(auto scripts will increase about 3% cpu utilization)
Win10-64 can get 9% (if guest service stable) or 15+% utilization(guest not idle) in host.

Can we accept the result(host utilization <10%) when running with automation?

Thanks
Yu Wang

(In reply to Yu Wang from comment #19)
> Tried on a clean win10-64 (1909)
> 
> If the guest cpu utilization is closed to 0% for a while, the host cpu is
> 4%-6%.
> But the service for win10 is more active than other guests. So, the result
> is 
> not as well as others. In another words ,I cannot hold the idle status for a
> long
> time. (sysMain service is always use the cpu)

sysmain is just a new name for Superfetch. MSFT used to recommend disabling it on
Hyper-V guest 
https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/role/hyper-v-server/processor-performance
I believe is what we should do as well.

> 
> tried with other guests (below win10), can get 4%-5%, since the guest status
> is stable.
> 
> Above result config:
> I closed the windows defender and windows updates, disable Automatic Windows
> Update, 
> disable NIC , unused USB ports, and CD-ROM. 


> 
> 
> And a question for my automation scripts:
> 
> I tried with my automation scripts.The result is:
> 
> Win8-32/win8.1-64,win2016,win2019 can get 7%-9% cpu utilization in host.
> (auto scripts will increase about 3% cpu utilization)
> Win10-64 can get 9% (if guest service stable) or 15+% utilization(guest not
> idle) in host.
> 
> Can we accept the result(host utilization <10%) when running with automation?

I think it can be acceptable, but definitely would prefer seeing it around 5%

Cheers,
Vadim.

> 
> Thanks
> Yu Wang

(In reply to Vadim Rozenfeld from comment #20)
> (In reply to Yu Wang from comment #19)
> > Tried on a clean win10-64 (1909)
> > 
> > If the guest cpu utilization is closed to 0% for a while, the host cpu is
> > 4%-6%.
> > But the service for win10 is more active than other guests. So, the result
> > is 
> > not as well as others. In another words ,I cannot hold the idle status for a
> > long
> > time. (sysMain service is always use the cpu)
> 
> sysmain is just a new name for Superfetch. MSFT used to recommend disabling
> it on
> Hyper-V guest 
> https://docs.microsoft.com/en-us/windows-server/administration/performance-
> tuning/role/hyper-v-server/processor-performance
> I believe is what we should do as well.
> 

Got it, I disabled this service and the utilization in guest is more stable than before.

And I have another question about cpu utilization with different vcpus:
(There are 48 vcpus in host, guest win10-64/ws2012/ws2019)

4 vcpus in guest, the utilization will be 7% in host
6 vcpus in guest, the utilization will be 9% in host
24 vcpus in guest,the utilization will be 30% in host

So the vcpu will influence the performance a lot.
When we test this case, will we restrict the quantity of guest vcpus?

(Note:above results are running with auto scripts, so the results will a bit more than 
"by manual")

Thanks
Yu Wang

(In reply to Yu Wang from comment #21)
> (In reply to Vadim Rozenfeld from comment #20)
> > (In reply to Yu Wang from comment #19)
> > > Tried on a clean win10-64 (1909)
> > > 
> > > If the guest cpu utilization is closed to 0% for a while, the host cpu is
> > > 4%-6%.
> > > But the service for win10 is more active than other guests. So, the result
> > > is 
> > > not as well as others. In another words ,I cannot hold the idle status for a
> > > long
> > > time. (sysMain service is always use the cpu)
> > 
> > sysmain is just a new name for Superfetch. MSFT used to recommend disabling
> > it on
> > Hyper-V guest 
> > https://docs.microsoft.com/en-us/windows-server/administration/performance-
> > tuning/role/hyper-v-server/processor-performance
> > I believe is what we should do as well.
> > 
> 
> Got it, I disabled this service and the utilization in guest is more stable
> than before.
> 
> And I have another question about cpu utilization with different vcpus:
> (There are 48 vcpus in host, guest win10-64/ws2012/ws2019)
> 
> 4 vcpus in guest, the utilization will be 7% in host
> 6 vcpus in guest, the utilization will be 9% in host
> 24 vcpus in guest,the utilization will be 30% in host
> 
> So the vcpu will influence the performance a lot.
> When we test this case, will we restrict the quantity of guest vcpus?

Do you use vCPU pinning? If not, vCPU may be scheduled across any of 
the cores in the system, which in turn can lead to not optimal cache
performance.

Cheers,
Vadim.

> 
> (Note:above results are running with auto scripts, so the results will a bit
> more than 
> "by manual")
> 
> Thanks
> Yu Wang

(In reply to Vadim Rozenfeld from comment #22)
> (In reply to Yu Wang from comment #21)

> > 
> > And I have another question about cpu utilization with different vcpus:
> > (There are 48 vcpus in host, guest win10-64/ws2012/ws2019)
> > 
> > 4 vcpus in guest, the utilization will be 7% in host
> > 6 vcpus in guest, the utilization will be 9% in host
> > 24 vcpus in guest,the utilization will be 30% in host
> > 
> > So the vcpu will influence the performance a lot.
> > When we test this case, will we restrict the quantity of guest vcpus?
> 
> Do you use vCPU pinning? If not, vCPU may be scheduled across any of 
> the cores in the system, which in turn can lead to not optimal cache
> performance.

Do you mean that adding "numactl --physcpubind=1,2,3,4,5,6 " in qemu cmd , 
is that enough or we have an other cpu pinning method ?

Thanks
Yu Wang

> 
> Cheers,
> Vadim.

Hi Vadim,

According to Comment#26 and Comment#28, we got less than 5% cpu utilization for each thread on win10 guests.

So, I cannot reproduce high cpu utilization in host, maybe this bug can be closed.


Thanks
Yu Wang

(In reply to Yu Wang from comment #31)
> Hi Vadim,
> 
> According to Comment#26 and Comment#28, we got less than 5% cpu utilization
> for each thread on win10 guests.
> 
> So, I cannot reproduce high cpu utilization in host, maybe this bug can be
> closed.
> 
> 
> Thanks
> Yu Wang

No objections. Let's close it,
but keeping eyes on new Windows releases.

Best,
Vadim.

(In reply to Vadim Rozenfeld from comment #32)
> (In reply to Yu Wang from comment #31)
> > Hi Vadim,
> > 
> > According to Comment#26 and Comment#28, we got less than 5% cpu utilization
> > for each thread on win10 guests.
> > 
> > So, I cannot reproduce high cpu utilization in host, maybe this bug can be
> > closed.
> > 
> > 
> > Thanks
> > Yu Wang
> 
> No objections. Let's close it,
> but keeping eyes on new Windows releases.

OK, got it.

Thanks
Yu Wang 
> 
> Best,
> Vadim.