Bug 889143
| Summary: | Guest with virtio-net-pci in openvswitch bridge, the net-speed is very slow between guests(in diffenent hosts) if configured vlan | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Qian Guo <qiguo> | ||||||||||
| Component: | qemu-kvm | Assignee: | Michael S. Tsirkin <mst> | ||||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||||
| Severity: | medium | Docs Contact: | |||||||||||
| Priority: | medium | ||||||||||||
| Version: | 6.4 | CC: | acathrow, areis, bsarathy, chayang, dyasny, juzhang, mkenneth, tgraf, virt-maint, xwei | ||||||||||
| Target Milestone: | rc | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2013-04-16 21:57:20 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
Created attachment 666580 [details]
test netperf with vlan
Created attachment 666581 [details]
test scp with vlan
Created attachment 666582 [details]
test netperf w/o vlan
Created attachment 666583 [details]
test scp w/o vlan
Test netperf with vlan in host level, the output is normal, info list below
1. host A:
#ifconfig eth1.10
eth1.10 Link encap:Ethernet HWaddr A0:36:9F:0F:7C:71
inet addr:192.168.10.11 Bcast:192.168.10.255 Mask:255.255.255.0
#ip -d link show
32: eth1.10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether a0:36:9f:0f:7c:71 brd ff:ff:ff:ff:ff:ff
vlan id 10 <REORDER_HDR>
2. host B:
#ifconfig eth1.10
eth1.10 Link encap:Ethernet HWaddr 00:15:17:DE:0C:87
inet addr:192.168.10.21 Bcast:192.168.10.255 Mask:255.255.255.0
#ip -d link show
41: eth1.10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 00:15:17:de:0c:87 brd ff:ff:ff:ff:ff:ff
vlan id 10 <REORDER_HDR>
3. host A run
#netsever
4. host B run
# netperf -H 192.168.10.11
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.11 () port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.03 938.69
so the network works well with vlan in host level
Windows 7 32bit guest has the same issue This request was not resolved in time for the current release. Red Hat invites you to ask your support representative to propose this request, if still desired, for consideration in the next release of Red Hat Enterprise Linux. Does the performance improve if you disable TSO in both the guest and the host? (In reply to comment #9) > Does the performance improve if you disable TSO in both the guest and the > host? Hi Thomas, I will update the result after I test w/ tso disable asap. (In reply to comment #9) > Does the performance improve if you disable TSO in both the guest and the > host? Test openvswitch-1.7.1-7.el6.x86_64 in rhel6.4 host The performance is good when disable TSO in the guest, which send pkg to the other. Details: Before test, check the info in GUEST guest# ethtool -k eth1 Features for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off rx-vlan-offload: off tx-vlan-offload: off ntuple-filters: off receive-hashing: off Both guests in vlan 10 1 Test netperf: 1). All guests and hosts without disable tso, one guest run netserver, and the other run: # netperf -H 10.0.0.11 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.11 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 13.02 0.10 2). Enable TSO of one guest, which run the netperf client: # ethtool -K eth1 tso off # ethtool -k eth1 Features for eth1: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: off udp-fragmentation-offload: on generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off rx-vlan-offload: off tx-vlan-offload: off ntuple-filters: off receive-hashing: off # netperf -H 10.0.0.11 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.11 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.03 933.27 So the send size is OK when disable TSO of the guest. 2. Test scp big file: 1) All guests and hosts without disable tso, from one guest to send big file to the other: # scp /home/bigfile root.0.11:/home/ root.0.11's password: bigfile 0% 2112KB 1.2MB/s - stalled - bigfile 0% 2112KB 909.2KB/s - stalled - bigfile 0% 2208KB 604.3KB/s 28:51 ETA bigfile 0% 2208KB 396.5KB/s - stalled - bigfile 0% 2208KB 234.1KB/s - stalled - bigfile 0% 2320KB 78.3KB/s - stalled - bigfile 0% 2448KB 33.6KB/s 8:38:30 ETA bigfile 0% 2448KB 9.5KB/s - stalled - bigfile 0% 2576KB 2.8KB/s - stalled - 2) Disable TSO of the guest,witch send pkgs, then send big file to the other guest: # scp /home/bigfile root.0.11:/home/ root.0.11's password: bigfile 100% 1024MB 41.0MB/s 00:25 The performance is ok and no need to disable hosts' TSO *** This bug has been marked as a duplicate of bug 915551 *** |
Description of problem: 2 rhel6.3 guests in different hosts with virtio-net-pci, configure their nics in same vlan(openvswitch based) the network would be very slow, 1.netperf TCP STREAM test result list below #netperf -H 192.168.10.21 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.21 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 12.99 0.10 2.scp test list below # scp bigfile root.10.21:/home/ bigfile 0% 2112KB 1.9MB/s 08:57 ETA bigfile 0% 2112KB 1.4MB/s 12:17 ETA bigfile 0% 2112KB 909.2KB/s - stalled - bigfile 0% 2112KB 736.4KB/s - stalled - bigfile 0% 2208KB 545.5KB/s 31:13 ETA bigfile 0% 2208KB 322.1KB/s - stalled - bigfile 0% 2208KB 190.2KB/s - stalled - bigfile 0% 2320KB 102.2KB/s 2:46:40 ETA bigfile 0% 2320KB 39.6KB/s - stalled - bigfile 0% 2320KB 17.0KB/s - stalled - bigfile 0% 2448KB 8.3KB/s - stalled - bigfile 0% 2576KB 3.5KB/s - stalled - bigfile 0% 2704KB 2.5KB/s - stalled - Version-Release number of selected component (if applicable): 1. host kernel version kernel-2.6.32-348.el6.x86_64 2. openvswitch version openvswitch-1.7.1-7.el6.x86_64 3. qemu-kvm-rhev version qemu-kvm-rhev-0.12.1.2-2.346.el6.x86_64 4. guest os:RHEL6.3, kernel:kernel-2.6.32-279.18.1.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1.run guest A in host a with virtio-net-pci, corresponding tap in any vlan (i use vlan 10) 2.run guest B in host b with virtio-net-pci, corresponding tap in the same vlan as guset A 3.configure ip addresses of A and B in same subnet 4 4.1.in A, run #netserver, in B, run #nerperf -H ${IP of B} 4.2.from A scp a big file(i use a 1G file) to B Actual results: the Throughput is too slow, and the "scp" is stalled Expected results: network works well Additional info: 1. These two hosts are both in same physical switch, and the switch ports are configured trunk mode. No speed limited in this switch. 2.I test similar scenario in one hosts(two guest both in one hosts), network is ok. 3.I test same scenario with e1000 and rtl8139, not hit this issue. 4.If not use vlan, these two guests' network is ok. 5.The guest cli like this #/usr/libexec/qemu-kvm -cpu SandyBridge -m 2G -smp 2,sockets=1,cores=2,threads=1 -M rhel6.4.0 -enable-kvm -name ovs2 -drive file=/home/rhel6u3uz0501.qcow2,if=none,format=qcow2,werror=stop,rerror=stop,cache=none,media=disk,id=drive-scsi0-disk0 -device virtio-scsi-pci,id=scsi0,addr=0x4 -device scsi-hd,scsi-id=0,lun=0,bus=scsi0.0,drive=drive-scsi0-disk0,id=virtio-disk0,bootindex=1 -nodefaults -nodefconfig -monitor stdio -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup_vlan10,downscript=/etc/ovs-ifdown -device virtio-net-pci,netdev=hostnet0,id=virt-net0,mac=54:52:1a:32:b4:62 -spice disable-ticketing,port=5901 -vga qxl I will add attachments for netperf/scp tests about both with vlan and not.