Bug 1237024

Summary: [virtio-win][netkvm]ipv6 uploading speed is quite slow when set "TCP/UDP checksum offload(IPv6)" to "Rx & Tx Enabled"
Product: Red Hat Enterprise Linux 7 Reporter: lijin <lijin>
Component: virtio-winAssignee: Yvugenfi <yvugenfi>
virtio-win sub component: virtio-win-prewhql QA Contact: Virtualization Bugs <virt-bugs>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jen, lmiksik, pl, vrozenfe, vyasevic, wyu, yvugenfi
Version: 7.2   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
NO_DOCS
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 08:47:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
"tcp/udp checksum offload of ipv6" is "rx/tx enabled"
none
"tcp/udp checksum offload of ipv6" is "rx enabled"
none
ipv6 packet capture none

Description lijin 2015-06-30 08:40:07 UTC
Description of problem:
when set netkvm parameter "TCP/UDP checksum offload(IPv6)" to the default value as "RX/TX enabled",speed of upload via ipv6 is quite slow.Change value to "Rx enabled" can resolve this issue

Version-Release number of selected component (if applicable):
virtio-win-prewhql-105
qemu-kvm-rhev-2.3.0-2.el7.x86_64
kernel-3.10.0-263.el7.x86_64
seabios-1.7.5-8.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot a win2012R2 guest with netkvm:
/usr/libexec/qemu-kvm -name 105NIC2008R2CL4 -enable-kvm -m 4G -smp 4 -uuid 8a45516d-caa5-4f34-940c-ce1b8c86fa7b -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/tmp/105NIC2008R2CL4,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=win2012R2.raw,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga cirrus -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet1,vhost=on,queues=4 -device virtio-net-pc
i,netdev=hostnet1,id=net1,mac=00:52:21:7f:13:03,bus=pci.0,mq=on,vectors=10
2.install netkvm driver and keep all parameters as default value
3.install winscp software
4.use winscp uploading a 3G file to host via ipv6
  winscp "hostname" field - host ipv6 address
5.set netkvm parameter "TCP/UDP checksum offload(IPv6)" to "RX enabled"
6.repeat step4

Actual results:
after step4,default value of "TCP/UDP checksum offload(IPv6)" is "RX/TX enabled";
the upload speed is less than 200Kib/s(please check the first attachment),it may take 15+hours to finish uploading,I havn't wait so long to check if upload can be finished correctly. 

after step6,"TCP/UDP checksum offload(IPv6) is set to "RX enabled",guest can upload files correctly ,and the speed is about 8000Kib/s(please check the second attachment)

Expected results:
guest can upload files normally via ipv6 when "TCP/UDP checksum offload(IPv6)" is "RX/TX enabled";

Additional info:

Comment 1 lijin 2015-06-30 08:41:13 UTC
Created attachment 1044620 [details]
"tcp/udp checksum offload of ipv6" is "rx/tx enabled"

Comment 2 lijin 2015-06-30 08:41:42 UTC
Created attachment 1044621 [details]
"tcp/udp checksum offload of ipv6" is "rx enabled"

Comment 4 Yvugenfi@redhat.com 2015-06-30 10:39:07 UTC

* When you experience the issues is it between VMs on the same host?

I can reproduce it between a Windows VM on Host A and a Linux VM on Host B.
Linux and Windows are not on the same host systems due to licensing.

Between Linux VMs and/or if I download to the Windows VM there is no issue.
Only outbound TCP traffic from the Windows vServer is affected and it only affects IPv6 traffic.


* Can you describe your host network configuration?

Windows VM on Qemu -> tap -> Linux Bridge -> eth (tagged) -> Network -> eth(tagged) -> Linux Bridge -> tap -> Linux VM on Qemu.


* Can provide the output for "ethtool -k” for every NIC (including tap devices) in the host that is involved with the VM that has a problem?

Offload parameters for tap4:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

Offload parameters for br136:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off




 * What are the host kernel version, QEMU version and guest driver version you are using?

host: 3.13.0-53-generic
qemu: 2.2.1
guest dirver: 0.1.100

Comment 5 Vlad Yasevich 2015-07-07 16:11:00 UTC
Interesting...

I've tried looking and something is going very wrong for me.

I have Windows Server 2012 R2 Build 9600 (installed from DVD iso
http://team.virt.bos.redhat.com/microsoft/)

I've loaded what I think is the right version of virtio-win from
https://brewweb.devel.redhat.com/buildinfo?buildID=438890.
Windows shows driver version 62.72.104.10500 from 6/3/2015.

When I try to upload a file regardless of IPv4 or IPv6, I see corrupted
LSO packet that is later retransmited, thus resulting in horrible throughput.

If I disable LSO, I get about a 2Gpbs upload speed.

I'll try to hold on to this machine if you want to take a look.

If I had to guess, there might be some kind of checksum interaction issues
when you turn off TX checksum offloading but leave LSO on.

-vlad

Comment 6 Vlad Yasevich 2015-07-07 21:09:50 UTC
Created attachment 1049571 [details]
ipv6 packet capture

I was able to reproduce the issue with ipv6 traffic, but for me it seems to happen always.

I've attached the tcpdump of the ipv6 traffic.  The large TCPv6 segments seem to
be corrupt/truncated.  The capture was taken on the vnet1 tap device.

-vlad

Comment 9 Yvugenfi@redhat.com 2015-07-19 08:44:58 UTC
(In reply to Vlad Yasevich from comment #6)
> Created attachment 1049571 [details]
> ipv6 packet capture
> 
> I was able to reproduce the issue with ipv6 traffic, but for me it seems to
> happen always.
> 
> I've attached the tcpdump of the ipv6 traffic.  The large TCPv6 segments
> seem to
> be corrupt/truncated.  The capture was taken on the vnet1 tap device.
> 
> -vlad

There was a bug in the driver.

Comment 10 Vadim Rozenfeld 2015-07-21 06:10:05 UTC
should be fixed in build 106 https://brewweb.devel.redhat.com//buildinfo?buildID=448339

Comment 11 lijin 2015-07-21 06:17:27 UTC
wyu,please verify this bug with build106

Comment 12 Yu Wang 2015-07-22 02:56:41 UTC
Reproduced this issue on virtio-win-prewhql-105 version
Verified this issue on virtio-win-prewhql-106 version
  
Steps same as comment#0

Actual Results:

on virtio-win-prewhql-105 (un-fixed version), the issue can be reproduced as comment#0
on virtio-win-prewhql-106 (fix version), guest can upload files normally via ipv6 when "TCP/UDP checksum offload(IPv6)" is "RX/TX enabled"
  
Base on above, this issue has been fixed already.



Version-Release number of selected component 
virtio-win-prewhql-106
qemu-kvm-rhev-2.3.0-12.el7.x86_64
kernel-3.10.0-296.el7.x86_64
seabios-1.7.5-9.el7.x86_64

Comment 13 lijin 2015-07-22 08:27:40 UTC
change status to verified according to comment#12

Comment 15 lijin 2015-10-22 02:24:16 UTC
This bug is fixed in virtio-win-prewhql-0.1-106,but as we discussed,we are not going to push virtio-win-prewhql-0.1-106 in rhel7.2,we push virtio-win-prewhql-0.1-105/110(build110 is also based on build105) instead.

So clear "Fixed In Version" field and this bug will not added into rhel7.2 errata.

Comment 19 errata-xmlrpc 2016-11-04 08:47:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2609.html