Description of problem: Enable netkvm driver TxLSO,there is no packet length >= 1514 after file transfer. Version-Release number of selected component (if applicable): kernel-5.14.0-174.el9.x86_64 qemu-kvm-7.1.0-2.el9.x86_64 edk2-ovmf-20220526git16779ede2d36-4.el9.noarch virtio-win-prewhql-0.1-227.iso How reproducible: always Steps to Reproduce: 1.Boot a windows guest /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ -blockdev node-name=file_ovmf_vars,driver=file,filename=/root/avocado/data/avocado-vt/avocado-vt-vm1_win2022-64-virtio-scsi_qcow2_filesystem_VARS.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 12000 \ -object memory-backend-ram,size=12000M,id=mem-machine_mem \ -smp 24,maxcpus=24,cores=12,threads=1,dies=1,sockets=2 \ -cpu 'Icelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,avx512ifma=on,sha-ni=on,rdpid=on,fsrm=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,mpx=off,intel-pt=off,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,kvm_pv_unhalt=on \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/win2022-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:68:f3:47:a3:b6,id=idu1RFWM,netdev=idTu1SNV,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idTu1SNV,vhost=on \ -blockdev node-name=file_cd1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \ -device scsi-cd,id=cd1,drive=drive_cd1,write-cache=on \ -blockdev node-name=file_virtio,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-227.iso,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_virtio,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_virtio \ -device scsi-cd,id=virtio,drive=drive_virtio,write-cache=on \ -vnc :0 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \ -monitor stdio \ 2.make sure netkvm verifier has been enabled 3.Install winpcap D:\AutoIt3_%PROCESSOR_ARCHITECTURE%.exe D:\install_winpcap.au3 4.Prepare the netkvmco.dll xcopy E:\NetKVM\2k22\amd64\netkvmco.dll c:\ /y && rundll32 netkvmco.dll,RegisterNetKVMNetShHelper 5.Enable scatter gather netsh netkvm setparam 0 param=Offload.TxLSO value=1 6.Restart nic to apply changes netsh interface set interface name="Ethernet Instance 0" admin=DISABLED netsh interface set interface name="Ethernet Instance 0" admin=ENABLED 7.Log network traffic with scatter gather enabled & Start wireshark session start "" "C:\Program Files\Wireshark\tshark.exe" -n -w c:\temp.pcapng tcp and dst 192.168.122.1 and src 192.168.122.95 tasklist /fi "IMAGENAME eq tshark.exe" 8.Start file transfer.Start file transfer --> Transferring file host -> guest 9.Stop wireshark taskkill /im tshark.exe /f 10.Parse wireshark log file "C:\Program Files\Wireshark\tshark.exe" -2 -r c:\temp.pcapng -R "frame.len>1514" No packet length >= 1514 Actual results: No packet length >= 1514 Expected results: enable netkvm driver TxLSO,some packets length should over 1514 Additional info: 1. Automation log: http://virtqetools.lab.eng.pek2.redhat.com/autotest_static_job_log/7105480/test-results/060-Host_RHEL.m9.u2.ovmf.qcow2.virtio_scsi.up.virtio_net.Guest.Win2022.x86_64.io-github-autotest-qemu.enable_scatter_windows.q35/ 2. qemu-kvm-7.1.0-1.el9.x86_64 also hit this problem
Test Matrix Test on the rhel.9.2.0 compose:http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-20221006.d.0/ 1. kernel-5.14.0-162.6.1.el9_1.x86_64 + qemu-kvm-7.1.0-2.el9.x86_64 -> Test Failed 2. qemu-kvm-7.0.0-13.el9.x86_64.rpm + kernel-5.14.0-174.el9.x86_64.rpm -> Test Failed 3. qemu-kvm-7.0.0-13.el9.x86_64.rpm + kernel-5.14.0-162.6.1.el9_1.x86_64 -> Test Failed Test on the rhel.9.1.0 compose: http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.1.0-20221012.1 1. qemu-kvm-7.0.0-13.el9.x86_64.rpm + kernel-5.14.0-162.6.1.el9_1.x86_64 -> Test Pass 2. qemu-kvm-7.1.0-2.el9.x86_64 + kernel-5.14.0-162.6.1.el9_1.x86_64 -> Test Pass 3. qemu-kvm-7.1.0-2.el9.x86_64 + kernel-5.14.0-174.el9.x86_64.rpm -> Test Pass Based on the above test result, seems this bug not releate to qemu and kernel, but it must be a product bug.Temporarily use this component to track the problem until the real cause is found
(In reply to Lei Yang from comment #1) ... > Based on the above test result, seems this bug not releate to qemu and > kernel, but it must be a product bug.Temporarily use this component to track > the problem until the real cause is found I think this is more related to Windows so I change the SST pool id.
(In reply to Laurent Vivier from comment #2) > (In reply to Lei Yang from comment #1) > ... > > Based on the above test result, seems this bug not releate to qemu and > > kernel, but it must be a product bug.Temporarily use this component to track > > the problem until the real cause is found > > I think this is more related to Windows so I change the SST pool id. Marek, could you confirm?
Hi Yan, can you take a look? Thanks!
What happens when the default values is not changed (should be on as well)?
(In reply to Yvugenfi from comment #5) Hello Yan > What happens when the default values is not changed (should be on as well)? What do you mean by default values? During the entire testing process, nothing seems to have changed. Thanks Lei
netsh netkvm setparam 0 param=Offload.TxLSO value=1 The default value in INF is 2. In any case, LSO should be enabled. Best regards, Yan.
Transfer guest file to host, then large packets make sense much more. After you try, please respond.
(In reply to ybendito from comment #8) > Transfer guest file to host, then large packets make sense much more. > After you try, please respond. Hello Yuri This step is already included in the current test log： vocado.virttest.virt_vm DEBUG| Attempting to log into 'avocado-vt-vm1' via serial console (timeout 360s) aexpect.client DEBUG| Sending command: start "" "C:\Program Files\Wireshark\tshark.exe" -n -w c:\temp.pcapng tcp and dst 192.168.122.1 and src 192.168.122.95 aexpect.client DEBUG| Sending command: echo %errorlevel% aexpect.client DEBUG| Sending command: tasklist /fi "IMAGENAME eq tshark.exe" aexpect.client DEBUG| Sending command: echo %errorlevel% avocado.test INFO | Context: Start file transfer avocado.virttest.virt_vm DEBUG| Attempting to log into 'avocado-vt-vm1' (timeout 360s) avocado.virttest.virt_vm DEBUG| Found/Verified IP 192.168.122.95 for VM avocado-vt-vm1 NIC 0 avocado.virttest.utils_test INFO | Context: Start file transfer --> Login to guest avocado.virttest.utils_test INFO | Context: Start file transfer --> Creating 50MB file on host avocado.utils.process INFO | Running 'dd if=/dev/zero of=/var/tmp/avocado_lut54745/tmp-epqz_1pi bs=10M count=5' avocado.utils.process DEBUG| [stderr] 5+0 records in avocado.utils.process INFO | Command 'dd if=/dev/zero of=/var/tmp/avocado_lut54745/tmp-epqz_1pi bs=10M count=5' finished with 0 after 0.023190531s avocado.utils.process DEBUG| [stderr] 5+0 records out avocado.utils.process DEBUG| [stderr] 52428800 bytes (52 MB, 50 MiB) copied, 0.0224352 s, 2.3 GB/s avocado.virttest.utils_test INFO | Context: Start file transfer --> Transferring file host -> guest, timeout: 1000s aexpect.remote DEBUG| Sending file /var/tmp/avocado_lut54745/tmp-epqz_1pi aexpect.remote INFO | Copy file from /var/tmp/avocado_lut54745/tmp-epqz_1pi to 192.168.122.95:c:\JciYIChB, elapsed time: 0.09739828109741211 avocado.virttest.utils_test INFO | Context: Start file transfer --> Transferring file guest -> host, timeout: 1000s aexpect.remote DEBUG| Receiving file /var/tmp/avocado_lut54745/tmp-epqz_1pi aexpect.remote INFO | Copy file from 192.168.122.95:c:\JciYIChB to /var/tmp/avocado_lut54745/tmp-epqz_1pi, elapsed time: 0.9929146766662598 avocado.virttest.utils_test INFO | Context: Start file transfer --> Compare md5sum between original file and transferred file avocado.virttest.utils_test INFO | Cleaning temp file on guest aexpect.client DEBUG| Sending command: del c:\JciYIChB aexpect.client DEBUG| Sending command: echo %errorlevel% avocado.test INFO | Context: Stop wireshark aexpect.client DEBUG| Sending command: taskkill /im tshark.exe /f aexpect.client DEBUG| Sending command: echo %errorlevel% avocado.test INFO | Context: Parse wireshark log file aexpect.client DEBUG| Sending command: "C:\Program Files\Wireshark\tshark.exe" -2 -r c:\temp.pcapng -R "frame.len>1514" aexpect.client DEBUG| Sending command: echo %errorlevel% avocado.test INFO | Check length > 1514 packets avocado.test ERROR| avocado.test ERROR| Reproduced traceback from: /usr/local/lib/python3.9/site-packages/avocado_framework_plugin_vt-98.0-py3.9.egg/avocado_vt/test.py:274 avocado.test ERROR| Traceback (most recent call last): avocado.test ERROR| File "/usr/local/lib/python3.9/site-packages/avocado_framework_plugin_vt-98.0-py3.9.egg/virttest/error_context.py", line 135, in new_fn avocado.test ERROR| return fn(*args, **kwargs) avocado.test ERROR| File "/root/avocado/data/avocado-vt/virttest/test-providers.d/downloads/io-github-autotest-qemu/qemu/tests/enable_scatter_windows.py", line 204, in run avocado.test ERROR| test.fail("No packet length >= 1514, output=%s" % output) avocado.test ERROR| File "/usr/local/lib/python3.9/site-packages/avocado_framework-98.0-py3.9.egg/avocado/core/test.py", line 779, in wrapper avocado.test ERROR| return func(actual_message) avocado.test ERROR| File "/usr/local/lib/python3.9/site-packages/avocado_framework-98.0-py3.9.egg/avocado/core/test.py", line 795, in fail avocado.test ERROR| raise exceptions.TestFail(msg) avocado.test ERROR| avocado.core.exceptions.TestFail: No packet length >= 1514, output= Thanks Lei
(In reply to Yvugenfi from comment #7) > netsh netkvm setparam 0 param=Offload.TxLSO value=1 > > The default value in INF is 2. In any case, LSO should be enabled. > > Best regards, > Yan. Hello Yan I tried to test this scenario with the latest compose 30 times,there is no issues any more.Since the compose mentioned in comment 1 is no longer available. Test Result: http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/?jobid=7173190 Based on the above test reuslt, seems this problem has been fixed, but I can not confirmed it. Therefore from the QE perspevtive,QE can continue to track this issue in subsequent tests, and if it has not been able to reproduce again, it can be closed as "CURRENTRELEASE". What do you think about it? Thanks in advance. Thanks Lei
(In reply to Lei Yang from comment #10) > (In reply to Yvugenfi from comment #7) > > netsh netkvm setparam 0 param=Offload.TxLSO value=1 > > > > The default value in INF is 2. In any case, LSO should be enabled. > > > > Best regards, > > Yan. > > Hello Yan > > I tried to test this scenario with the latest compose 30 times,there is no > issues any more.Since the compose mentioned in comment 1 is no longer > available. > > Test Result: > http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/ > ?jobid=7173190 > > Based on the above test reuslt, seems this problem has been fixed, but I can > not confirmed it. Therefore from the QE perspevtive,QE can continue to track > this issue in subsequent tests, and if it has not been able to reproduce > again, it can be closed as "CURRENTRELEASE". What do you think about it? > Thanks in advance. > > Thanks > Lei Hi Lei, Good idea. Let's check with subsequent tests as well. In any case, there were no changes in the area of configuration or LSO support in the driver recently. Best regards, Yan.
(In reply to Lei Yang from comment #10) > (In reply to Yvugenfi from comment #7) > > netsh netkvm setparam 0 param=Offload.TxLSO value=1 > > > > The default value in INF is 2. In any case, LSO should be enabled. > > > > Best regards, > > Yan. > > Hello Yan > > I tried to test this scenario with the latest compose 30 times,there is no > issues any more.Since the compose mentioned in comment 1 is no longer > available. > > Test Result: > http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/ > ?jobid=7173190 > > Based on the above test reuslt, seems this problem has been fixed, but I can > not confirmed it. Therefore from the QE perspevtive,QE can continue to track > this issue in subsequent tests, and if it has not been able to reproduce > again, it can be closed as "CURRENTRELEASE". What do you think about it? > Thanks in advance. Do you reproduce the problem anymore? Can we close the BZ?
(In reply to Laurent Vivier from comment #12) > (In reply to Lei Yang from comment #10) > > (In reply to Yvugenfi from comment #7) > > > netsh netkvm setparam 0 param=Offload.TxLSO value=1 > > > > > > The default value in INF is 2. In any case, LSO should be enabled. > > > > > > Best regards, > > > Yan. > > > > Hello Yan > > > > I tried to test this scenario with the latest compose 30 times,there is no > > issues any more.Since the compose mentioned in comment 1 is no longer > > available. > > > > Test Result: > > http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/ > > ?jobid=7173190 > > > > Based on the above test reuslt, seems this problem has been fixed, but I can > > not confirmed it. Therefore from the QE perspevtive,QE can continue to track > > this issue in subsequent tests, and if it has not been able to reproduce > > again, it can be closed as "CURRENTRELEASE". What do you think about it? > > Thanks in advance. > Hello Laurent > Do you reproduce the problem anymore? > Can we close the BZ? I tried to test this scenarios with latest compose 30 times, there is no issues any more. So this bug should be closed as "CURRENTRELEASE", please correct me if I'm wrong. Test Version: http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-20221107.5/ Test Log: http://virtqetools.lab.eng.pek2.redhat.com/kvm_autotest_job_log/?jobid=7224375
en, I am a little confused, this problem is reproduced again on the latest rhel 9.3. But not sure what caused this problem, temporarily use this bug to record this problem. Test Version: kernel-5.14.0-300.el9.x86_64 qemu-kvm-8.0.0-1.el9.x86_64 edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch virtio-win-prewhql-0.1-235.iso
(In reply to Lei Yang from comment #14) > en, I am a little confused, this problem is reproduced again on the latest > rhel 9.3. But not sure what caused this problem, temporarily use this bug to > record this problem. > > Test Version: > kernel-5.14.0-300.el9.x86_64 > qemu-kvm-8.0.0-1.el9.x86_64 > edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch > virtio-win-prewhql-0.1-235.iso Hi Lei, If there is a problem, I suggest reopening a bug or filing a new bug. Otherwise, we might lose the discussion. Again, there were no guest driver changes related to LSO. So if opening new bug the component should be qemu-kvm Best regards, Yan.
(In reply to Yvugenfi from comment #15) > (In reply to Lei Yang from comment #14) > > en, I am a little confused, this problem is reproduced again on the latest > > rhel 9.3. But not sure what caused this problem, temporarily use this bug to > > record this problem. > > > > Test Version: > > kernel-5.14.0-300.el9.x86_64 > > qemu-kvm-8.0.0-1.el9.x86_64 > > edk2-ovmf-20230301gitf80f052277c8-2.el9.noarch > > virtio-win-prewhql-0.1-235.iso > > Hi Lei, > > If there is a problem, I suggest reopening a bug or filing a new bug. > Otherwise, we might lose the discussion. > Again, there were no guest driver changes related to LSO. > So if opening new bug the component should be qemu-kvm > > Best regards, > Yan. Hello Yan Thanks for your update, let me reopen this bug. Thanks Lei
Hi Yan I review this bug again, from the QE's perspective this bug is not related qemu-kvm and virtio-win-driver. Because the checkpoint is: the scp file successfully and length of some package SSH protocol >=1514. It looks like there are some issues about openssh, what do you think about it? Thanks Lei
(In reply to Lei Yang from comment #18) > Hi Yan > > I review this bug again, from the QE's perspective this bug is not related > qemu-kvm and virtio-win-driver. Because the checkpoint is: the scp file > successfully and length of some package SSH protocol >=1514. It looks like > there are some issues about openssh, what do you think about it? > > Thanks > Lei Hi Lei, I am not sure. I might be related to the network configuration on the host (in virtio-net driver on the guest we don't have any related changes between the versions). So it might point to some potential problem that can affect the performance. Best regards, Yan.
Hi Wenli Could you please help review this bug? Maybe this is a potential problem that can affect the performance. Thanks in advance. Test Steps: 1. Boot a guest with virtio-net-pci 2. Change scatter-gather to enable for windows: enable Offload Tx LSO: device manager--> click Network adapters--> right-click Properties of Redhat VirtIO Ethernet Adapter---> Properties---> Advanced-->change"Offload Tx LSO" to enable 4. copy file from guest to host For windows: use winscp to scp file from guest to host ,and the same time ,open wireshark to listening the package. 5. After steps 4 For windows: host can receive the scp file successfully and length of some package wih SSH protocal >=1514 Thanks Lei
QE tested bug Comment 0 steps again,this time QE noticed that c:\temp.pcapnp was empty when the problem reproduced. But at this time the file is transferred, so this may be a problem with the tool itself. Please correct me if I'm wrong.
(In reply to Lei Yang from comment #20) > Hi Wenli > > Could you please help review this bug? Maybe this is a potential problem > that can affect the performance. Thanks in advance. > > Test Steps: > 1. Boot a guest with virtio-net-pci > 2. Change scatter-gather to enable > for windows: > enable Offload Tx LSO: > device manager--> click Network adapters--> right-click Properties of Redhat > VirtIO Ethernet Adapter---> Properties---> Advanced-->change"Offload Tx LSO" > to enable > 4. copy file from guest to host > For windows: use winscp to scp file from guest to host ,and the same time > ,open wireshark to listening the package. > 5. After steps 4 > For windows: host can receive the scp file successfully and length of some > package wih SSH protocal >=1514 > > Thanks > Lei Manually test it, the LSO on windows works correctly. with enabled lso, the len of packet is 16474. 3200 38.142872 10.73.225.119 → 10.73.225.50 SSHv2 16474 Server: Encrypted packet (len=16420) 3201 38.143344 10.73.225.119 → 10.73.225.50 SSHv2 16474 Server: Encrypted packet (len=16420) With disabled lso, the len of packet is 1000. 28485 9.420404 10.73.225.119 → 10.73.225.50 SSH 1000 Server: Encrypted packet (len=946) 28486 9.420424 10.73.225.119 → 10.73.225.50 SSH 1000 Server: Encrypted packet (len=946)