Bug 1708065
| Summary: | Enabling TSO(default value) in guest with e1000e nic, the guest network performance over ipv6 is poor (copy file from guest to host, only reach 2.2MB/s and even lower) | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Pei Zhang <pezhang> | |
| Component: | qemu-kvm | Assignee: | Yvugenfi <yvugenfi> | |
| qemu-kvm sub component: | Networking | QA Contact: | Lei Yang <leiyang> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | low | |||
| Priority: | low | CC: | ailan, chayang, ddepaula, jasowang, jinzhao, juzhang, leiyang, mdean, rbalakri, virt-maint, ybendito, yvugenfi | |
| Version: | 8.1 | Keywords: | Triaged | |
| Target Milestone: | rc | |||
| Target Release: | 8.4 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | qemu-kvm-5.2.0-5.module+el8.4.0+9775+0937c167 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1708120 (view as bug list) | Environment: | ||
| Last Closed: | 2021-05-25 06:41:20 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1708120 | |||
Probably because of a buggy TSO implementation for e1000e. Does disable TSO in guest help? Can you try upstream qemu? Anyway, very low priority. Thanks (In reply to jason wang from comment #1) > Probably because of a buggy TSO implementation for e1000e. Does disable TSO > in guest help? Can you try upstream qemu? Hi Jason, Thanks for your immediately response. Disable TSO in guest, the the speed is acceptable, it's 114.6MB/s # ethtool -K ens2 tso off # ethtool -k ens2 Features for ens2: ... tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: off ... Thanks, Pei > Anyway, very low priority. > > Thanks Additional info(continued): 3. Copy file with e1000e ovs ipv4 addr works well. guest->host: 174.0MB/s host->guest: 165.7MB/s (In reply to Pei Zhang from comment #2) > (In reply to jason wang from comment #1) > > Probably because of a buggy TSO implementation for e1000e. Does disable TSO > > in guest help? Can you try upstream qemu? Tried upstream qemu: git://git.qemu.org/qemu.git, master branch. Hit same issue, too. Keep default TSO on, the speed is around 2MB/s. Disable TSO, the speed is around 110MB/s. Thanks, Pei QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks https://www.mail-archive.com/qemu-devel@nongnu.org/msg723411.html https://www.mail-archive.com/qemu-devel@nongnu.org/msg723412.html Yan, please backport the fix if this is intended to arrive in RHEL 8.3 and also set the ITR field. If not, then please include which version of upstream qemu contains this fix and include the commit ID in the whiteboard. The patches are in version 5.1: 9a8d949245a0a3f90842d1611f56df6ae705560d - hw/net: Added CSO for IPv e219d30910d8584868dd5c11df10ba2a1336034c - hw/net: Added plen fix for IPv6 Sorry, actually additional commit is queued for 5.2 by Jason: https://www.mail-archive.com/qemu-devel@nongnu.org/msg723712.html Hi, Yan According to comment 15,I have some questions to confirm with you: 1.Do we have plans to fix this bug on RHEL.8.4? If it will be fixed on RHEL.8.4, can I temporarily set it to ITM16？ 2.Or the stale date should be reset? Best regrads Lei Yang (In reply to Lei Yang from comment #16) > Hi, Yan > > According to comment 15,I have some questions to confirm with you: > 1.Do we have plans to fix this bug on RHEL.8.4? If it will be fixed on > RHEL.8.4, can I temporarily set it to ITM16？ > 2.Or the stale date should be reset? > > Best regrads > Lei Yang Hi Lei Yang, The patches already in the upstream, so I expect to have the fix as part of RHEL8.4. Best regards, Yan. (In reply to Yan Vugenfirer from comment #17) > (In reply to Lei Yang from comment #16) > > Hi, Yan > > > > According to comment 15,I have some questions to confirm with you: > > 1.Do we have plans to fix this bug on RHEL.8.4? If it will be fixed on > > RHEL.8.4, can I temporarily set it to ITM16？ > > 2.Or the stale date should be reset? > > > > Best regrads > > Lei Yang > > Hi Lei Yang, > > The patches already in the upstream, so I expect to have the fix as part of > RHEL8.4. > > Best regards, > Yan. Thanks Yan, based on above setting ITM=16. Hit same issue Test Version: qemu-kvm-4.2.0-40.module+el8.4.0+9278+dd53883d.x86_64 kernel-4.18.0-269.el8.dt3.x86_64 (In reply to Lei Yang from comment #19) > Hit same issue > Test Version: > qemu-kvm-4.2.0-40.module+el8.4.0+9278+dd53883d.x86_64 > kernel-4.18.0-269.el8.dt3.x86_64 The changes in QEMU 5.2. Can you test please on QEMU 5.2? Thanks, Yan. (In reply to Yan Vugenfirer from comment #21) > Can you test please on QEMU 5.2? > > Thanks, > Yan. Hi Yan I tried test it with qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 =>Windows10 guest: e1000e: host -> guest: it's speed is around 136MB/s guest -> host: itls speed is around 400MB/s =>RHEl.8.4 guest: e1000e device: host -> guest: it's speed is around 97MB/s guest -> host: it's speed is around 126MB/s Best Regards Lei (In reply to Lei Yang from comment #22) > (In reply to Yan Vugenfirer from comment #21) > > Can you test please on QEMU 5.2? > > > > Thanks, > > Yan. > > Hi Yan > > I tried test it with qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 > =>Windows10 guest: > > e1000e: > host -> guest: it's speed is around 136MB/s > guest -> host: itls speed is around 400MB/s > > =>RHEl.8.4 guest: > e1000e device: > host -> guest: it's speed is around 97MB/s > guest -> host: it's speed is around 126MB/s > > > Best Regards > Lei Hi Lei, I think that the original problem is solved (bad performance with guest-> host when using IPv6): guest->host e1000e 2.2MB/s But there is a difference in performance between Windows and Linux VMs. I suggest using iperf or netperf to check that this is not scp implementation issue, and if there is still a similar difference in performance between Linux and Windows to open a new BZ. Thanks, Yan. (In reply to Yan Vugenfirer from comment #23) > (In reply to Lei Yang from comment #22) > > (In reply to Yan Vugenfirer from comment #21) > > > Can you test please on QEMU 5.2? > > > > > > Thanks, > > > Yan. > > > > Hi Yan > > > > I tried test it with qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 > > =>Windows10 guest: > > > > e1000e: > > host -> guest: it's speed is around 136MB/s > > guest -> host: itls speed is around 400MB/s > > > > =>RHEl.8.4 guest: > > e1000e device: > > host -> guest: it's speed is around 97MB/s > > guest -> host: it's speed is around 126MB/s > > > > > > Best Regards > > Lei > > Hi Lei, > > I think that the original problem is solved (bad performance with guest-> > host when using IPv6): > > guest->host > e1000e 2.2MB/s > > But there is a difference in performance between Windows and Linux VMs. > I suggest using iperf or netperf to check that this is not scp > implementation issue, and if there is still a similar difference in > performance between Linux and Windows to open a new BZ. > > Thanks, > Yan. Hi Yan I used iperf to check this problem. There are some differences between Windows and Linux, but I am not sure if this is acceptable, if this is a problem, please let me know that I will file a new bug. =>Windows guest: Guest as server： iperf.-2.0.12.exe -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 324] local 10.73.227.145 port 5001 connected with 10.73.224.24 port 59310 [ ID] Interval Transfer Bandwidth [ 324] 0.0-60.0 sec 2.56 GBytes 367 Mbits/sec Host as client: # iperf -c 10.73.227.145 -t 60 ------------------------------------------------------------ Client connecting to 10.73.227.145, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.73.224.24 port 59310 connected with 10.73.227.145 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 6.56 GBytes 939 Mbits/sec =>Linux guest Guest as server: # iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.73.227.111 port 5001 connected with 10.73.224.24 port 47478 [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec Host as client: # iperf -c 10.73.227.111 -t 60 ------------------------------------------------------------ Client connecting to 10.73.227.111, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.73.224.24 port 47478 connected with 10.73.227.111 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec Best Regards Lei (In reply to Lei Yang from comment #24) > (In reply to Yan Vugenfirer from comment #23) > > (In reply to Lei Yang from comment #22) > > > (In reply to Yan Vugenfirer from comment #21) > > > > Can you test please on QEMU 5.2? > > > > > > > > Thanks, > > > > Yan. > > > > > > Hi Yan > > > > > > I tried test it with qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 > > > =>Windows10 guest: > > > > > > e1000e: > > > host -> guest: it's speed is around 136MB/s > > > guest -> host: itls speed is around 400MB/s > > > > > > =>RHEl.8.4 guest: > > > e1000e device: > > > host -> guest: it's speed is around 97MB/s > > > guest -> host: it's speed is around 126MB/s > > > > > > > > > Best Regards > > > Lei > > > > Hi Lei, > > > > I think that the original problem is solved (bad performance with guest-> > > host when using IPv6): > > > > guest->host > > e1000e 2.2MB/s > > > > But there is a difference in performance between Windows and Linux VMs. > > I suggest using iperf or netperf to check that this is not scp > > implementation issue, and if there is still a similar difference in > > performance between Linux and Windows to open a new BZ. > > > > Thanks, > > Yan. > > Hi Yan > > I used iperf to check this problem. There are some differences between > Windows and Linux, but I am not sure if this is acceptable, if this is a > problem, please let me know that I will file a new bug. > > =>Windows guest: > Guest as server： > iperf.-2.0.12.exe -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 324] local 10.73.227.145 port 5001 connected with 10.73.224.24 port 59310 > [ ID] Interval Transfer Bandwidth > [ 324] 0.0-60.0 sec 2.56 GBytes 367 Mbits/sec > > Host as client: > # iperf -c 10.73.227.145 -t 60 > ------------------------------------------------------------ > Client connecting to 10.73.227.145, TCP port 5001 > TCP window size: 85.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.73.224.24 port 59310 connected with 10.73.227.145 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-60.0 sec 6.56 GBytes 939 Mbits/sec > > =>Linux guest > Guest as server: > # iperf -s > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 85.3 KByte (default) > ------------------------------------------------------------ > [ 4] local 10.73.227.111 port 5001 connected with 10.73.224.24 port 47478 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec > > Host as client: > # iperf -c 10.73.227.111 -t 60 > ------------------------------------------------------------ > Client connecting to 10.73.227.111, TCP port 5001 > TCP window size: 85.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.73.224.24 port 47478 connected with 10.73.227.111 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec > > Best Regards > Lei Hi Lei, It looks OK in regards to the original problem. Without working\configured RSC, I would expect Windows guest receive to work worse than Linux one. An in terms of performance parity our priority is virtio-net. Best regards, Yan. (In reply to Yan Vugenfirer from comment #25) > (In reply to Lei Yang from comment #24) > > (In reply to Yan Vugenfirer from comment #23) > > > (In reply to Lei Yang from comment #22) > > > > (In reply to Yan Vugenfirer from comment #21) > > > > > Can you test please on QEMU 5.2? > > > > > > > > > > Thanks, > > > > > Yan. > > > > > > > > Hi Yan > > > > > > > > I tried test it with qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08.x86_64 > > > > =>Windows10 guest: > > > > > > > > e1000e: > > > > host -> guest: it's speed is around 136MB/s > > > > guest -> host: itls speed is around 400MB/s > > > > > > > > =>RHEl.8.4 guest: > > > > e1000e device: > > > > host -> guest: it's speed is around 97MB/s > > > > guest -> host: it's speed is around 126MB/s > > > > > > > > > > > > Best Regards > > > > Lei > > > > > > Hi Lei, > > > > > > I think that the original problem is solved (bad performance with guest-> > > > host when using IPv6): > > > > > > guest->host > > > e1000e 2.2MB/s > > > > > > But there is a difference in performance between Windows and Linux VMs. > > > I suggest using iperf or netperf to check that this is not scp > > > implementation issue, and if there is still a similar difference in > > > performance between Linux and Windows to open a new BZ. > > > > > > Thanks, > > > Yan. > > > > Hi Yan > > > > I used iperf to check this problem. There are some differences between > > Windows and Linux, but I am not sure if this is acceptable, if this is a > > problem, please let me know that I will file a new bug. > > > > =>Windows guest: > > Guest as server： > > iperf.-2.0.12.exe -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 64.0 KByte (default) > > ------------------------------------------------------------ > > [ 324] local 10.73.227.145 port 5001 connected with 10.73.224.24 port 59310 > > [ ID] Interval Transfer Bandwidth > > [ 324] 0.0-60.0 sec 2.56 GBytes 367 Mbits/sec > > > > Host as client: > > # iperf -c 10.73.227.145 -t 60 > > ------------------------------------------------------------ > > Client connecting to 10.73.227.145, TCP port 5001 > > TCP window size: 85.0 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.73.224.24 port 59310 connected with 10.73.227.145 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-60.0 sec 6.56 GBytes 939 Mbits/sec > > > > =>Linux guest > > Guest as server: > > # iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 85.3 KByte (default) > > ------------------------------------------------------------ > > [ 4] local 10.73.227.111 port 5001 connected with 10.73.224.24 port 47478 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec > > > > Host as client: > > # iperf -c 10.73.227.111 -t 60 > > ------------------------------------------------------------ > > Client connecting to 10.73.227.111, TCP port 5001 > > TCP window size: 85.0 KByte (default) > > ------------------------------------------------------------ > > [ 3] local 10.73.224.24 port 47478 connected with 10.73.227.111 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-60.0 sec 10.8 GBytes 1.55 Gbits/sec > > > > Best Regards > > Lei > > Hi Lei, > > It looks OK in regards to the original problem. Without working\configured > RSC, I would expect Windows guest receive to work worse than Linux one. > An in terms of performance parity our priority is virtio-net. > > Best regards, > Yan. Hi Yan I think so too, this bug should be fixed well in qemu-kvm-5.2. Best Regards Lei Hi Lei, Please change BZ status to an appropriate one based on comment #26. Thanks, Yan. (In reply to Yan Vugenfirer from comment #27) > Hi Lei, > > Please change BZ status to an appropriate one based on comment #26. > > Thanks, > Yan. Hi Danilo Could you please help update to the errata? Best Regards Lei (In reply to Yan Vugenfirer from comment #27) > Hi Lei, > > Please change BZ status to an appropriate one based on comment #26. > > Thanks, > Yan. Hi Yan Based on bz workflow, the bz should be changed to modified and then we can request danilo to change to ON_QA, so could you change the bz status to modified? Best Regards Lei Hi Danilo Could you please help update to the errata? Best Regards Lei Changing to ON_QA and adding TestOnly keyword as this didn't required any code change. Hence the TestOnly flag, this will not be added to the 8.4.0 errata. Hi Danilo I'd like to remove "TestOnly" as actually there is code delivery in qemu(see comment 20). And QE confirmed the fix is in qemu-5.2 after rebase. Best Regards Lei Based on comments 22,23,24,25,26. This bug should be fixed well in qemu-kvm-5.2. So move to "VERIFIED". Done. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098 |
Description of problem: Boot guest with e1000e, then copy a big file from guest to host. The transfer speed is only around 2.2MB/s. However with virtio-net-pci, the transfer speed is around 300MB/s. guest->host host->guest e1000e 2.2MB/s 150MB/s virtio-net-pci 300MB/s 300MB/s Version-Release number of selected component (if applicable): 4.18.0-82.el8.x86_64 qemu-kvm-2.12.0-70.module+el8.1.0+3165+631e0cd3.x86_64 How reproducible: 100% Steps to Reproduce: 1. Boot guest with e1000e -netdev tap,id=hostnet0 \ -device e1000e,netdev=hostnet0,id=net0,mac=fa:5e:07:b4:08:01,bus=root.2 \ 2. DD a 4G file in guest # dd if=/dev/zero of=/root/bigfile oflag=direct bs=1M count=4096 3. Copy this 4G file from guest to host, the speed is much low, only around 2.2MB/s. # scp -r /root/bigfile root@\[fe80::b3df:567e:33b8:286%ens2\]:/root/ bigfile 1% 44MB 2.2MB/s 31:07 ETA Actual results: The transfer speed from guest to host over ipv6 addr is much lower 2MB/s, which is not acceptable. Expected results: The transfer speed from guest to host over ipv6 addr should be higher, maybe around 300MB/s. Additional info: 1. virtio-net-pci works well, it's speed is around 300MB/s. bigfile 71% 2921MB 305.9MB/s 00:03 ETA 2. With e1000e, copy file from host to guest, the speed is around 150MB/s. bigfile 39% 1620MB 149.6MB/s 00:16 ETA Reference: Full cmd line: /usr/libexec/qemu-kvm -name rhel8.0 \ -M q35,kernel-irqchip=split \ -cpu host -m 8G \ -smp 4,sockets=1,cores=4,threads=1 \ -device pcie-root-port,id=root.1,slot=1 \ -device pcie-root-port,id=root.2,slot=2 \ -drive file=/home/rhel8.0.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.1 \ -vnc :2 \ -monitor stdio \ -netdev tap,id=hostnet0 \ -device e1000e,netdev=hostnet0,id=net0,mac=fa:5e:07:b4:08:01,bus=root.2 \