Bug 2079670

Summary: Failed to run netperf tool with "-c" parameter when boot a win2022 guest with "-smp > 32"
Product: Red Hat Enterprise Linux 9 Reporter: Yiqian Wei <yiwei>
Component: virtio-winAssignee: Marek Kedzierski <mkedzier>
virtio-win sub component: others QA Contact: Wenkang Ji <wji>
Status: NEW --- Docs Contact:
Severity: high    
Priority: unspecified CC: coli, jinzhao, juzhang, leidwang, leiyang, mkedzier, nilal, virt-maint, vkuznets, vrozenfe, wji, xiagao
Version: 9.1Flags: leidwang: needinfo? (mkedzier)
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yiqian Wei 2022-04-28 04:11:29 UTC
Description of problem:
In win2022 guest, run netperf tool with " netperf.exe -H $host_ip -l 120 -C -c -t TCP_CRR " cmd, the netperf tool quit directly and prompts the following information:
NtQuery didn't return expected amount of data
Expected data for 32 CPUs, returned 33

Version-Release number of selected component (if applicable):
host version:
kernel-5.14.0-78.el9.x86_64
qemu-kvm-7.0.0-1.el9.x86_64
edk2-ovmf-20220221gitb24306f15d-1.el9.noarch
netperf-2.7.1.tar.bz2 
guest: win2022

How reproducible:
3/3

Steps to Reproduce:
1.Run netserver on host
# tar jxf netperf-2.7.1.tar.bz2 
# cd netperf-2.7.1  
# ./autogen.sh
# ./configure --build=x86_64  
# make
# cd netperf-2.7.1/src
# netserver

2.boot a win2022 guest with "-smp > 32"

3.Run netperf client on win2022 guest
# netperf.exe -H $host_ip -l 120 -C -c -t TCP_MAERTS

Actual results: the netperf tool quit directly and prompts the following information:
NtQuery didn't return expected amount of data
Expected data for 32 CPUs, returned 33

Expected results: netperf start successfully.

Additional info:
1) Boot a win2022 guest with "-smp <= 32", not hit this issue.
2) For rhel 9.1.0 guest, not hit this issue.
3) Not hit this issue when run netperf tool without "-c" parameter
# netperf.exe -H $host_ip -l 120 -C -t TCP_MAERTS
4)  host cpu: 
CPU(s):                  128
  On-line CPU(s) list:   0-127
Vendor ID:               AuthenticAMD
  BIOS Vendor ID:        AMD
  Model name:            AMD EPYC 7502 32-Core Processor
    BIOS Model name:     AMD EPYC 7502 32-Core Processor                
    CPU family:          23
    Model:               49
    Thread(s) per core:  2
    Core(s) per socket:  32
    Socket(s):           2

5) boot a win2022 guest with cli:
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/OVMF_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 62464 \
    -object memory-backend-ram,size=62464M,id=mem-machine_mem  \
    -smp 64,maxcpus=64,cores=32,threads=1,dies=1,sockets=2  \
    -cpu 'EPYC-Rome',x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,spec-ctrl=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,cmp-legacy=on,ibrs=on,amd-ssbd=on,virt-ssbd=on,svme-addr-chk=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/win2022-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:b5:5d:f0:c9:19,id=idAducli,netdev=idwQN2hl,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idwQN2hl,vhost=on  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \

Comment 10 Marek Kedzierski 2022-07-27 11:01:40 UTC
In short:
Netperf tool uses older Windows API to get the number of CPUs and undocumented API to 
get information about the performance of each CPU. Because older API has a limitation 
- if there are over 32 CPUs it always returns 32, I fixed the problem by using a 
newer API function to read the number of CPUs. To keep backward compatibility, 
the newer function is used only when is available.

Pool request for netperf is here:

https://github.com/HewlettPackard/netperf/pull/69

Comment 11 xiagao 2022-12-06 08:59:41 UTC
Still hit this issue on virtio-win-prewhql-0.1-230.

kernel-5.14.0-205.el9.x86_64
qemu-kvm-7.1.0-5.el9.x86_64
seabios-bin-1.16.0-4.el9.noarch
edk2-ovmf-20220826gitba0e0e4c6a-2.el9.noarch
virtiofsd-1.4.0-1.el9.x86_64

Comment 12 Marek Kedzierski 2022-12-06 09:10:41 UTC
(In reply to xiagao from comment #11)
> Still hit this issue on virtio-win-prewhql-0.1-230.
> 
> kernel-5.14.0-205.el9.x86_64
> qemu-kvm-7.1.0-5.el9.x86_64
> seabios-bin-1.16.0-4.el9.noarch
> edk2-ovmf-20220826gitba0e0e4c6a-2.el9.noarch
> virtiofsd-1.4.0-1.el9.x86_64

Hi, 

Did you get a fixed version of netperf from my repo?

Thanks, 

Marek

Comment 20 leidwang@redhat.com 2023-03-23 03:21:04 UTC
Hi Marek.

As we discussed before,we also need to fix this issue on netserver.exe.

Do you have any update about this bug? Thanks in advance!

Thanks,
Leidong

Comment 21 leidwang@redhat.com 2023-04-18 03:15:41 UTC
Hi Marek,

Could you please setup ITR and DTM for this bug?

Thanks,
Leidong

Comment 22 Wenkang Ji 2023-05-23 08:27:36 UTC
Still hit this issue on virtio-win-prewhql-0.1-236.

kernel-5.14.0-284.11.1.el9_2.x86_64
qemu-kvm-7.2.0-14.el9_2.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20221207gitfff6d81270b5-9.el9_2.noarch
virtiofsd-1.5.0-1.el9.x86_64