Hide Forgot
Description of problem: We're experiencing relatively slow network performance under KVM guests running Windows 2012 and Windows 2008. We are not seeing this problem with Linux guests nevertheless. I suspect it might be due to virtio drivers. The VMs are running on on 64-core AMD Opteron(tm) Processor 6386 SE CPU, and are connected to the network through Mellanox Technologies MT26428 Infiniband cards. Version-Release number of selected component (if applicable): virtio-win: 62.73.104.11200 Host distro: Debian Wheezy (with custom built 3.12.45 kernel) Qemu version: 2.0.2 Qemu command line: /usr/bin/kvm -uuid 91689c36-c8f6-4244-8841-d596274ed847 \ -m 2048 \ -M pc-1.0 \ -enable-kvm \ -nodefconfig \ -nodefaults \ -rtc base=utc \ -netdev tap,ifname=n02015d6f3294,id=hostnet6,script=no,downscript=no \ -device virtio-net-pci,netdev=hostnet6,id=net6,mac=02:01:5d:6f:32:94,bus=pci.0,addr=0x6 \ -netdev tap,ifname=n0201d4f27e6b,id=hostnet7,script=no,downscript=no \ -device virtio-net-pci,netdev=hostnet7,id=net7,mac=02:01:d4:f2:7e:6b,bus=pci.0,addr=0x7 \ -chardev pty,id=charserial0 \ -device isa-serial,chardev=charserial0,id=serial0 \ -usb \ -device usb-tablet,id=input0 \ -vnc 0.0.0.0:98 \ -vga std \ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 \ -cpu Opteron_G4,+cmp_legacy,+mmxext,+nrip_save,+osvw,+vme,+npt \ -smp 2,sockets=32,cores=1,maxcpus=64,threads=1 \ -drive file=uuid://6a47bff5-59b6-41fb-8ed1-91324ecf12b3,if=none,id=drive-virtio-disk5,format=raw,cache=none \ -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk5,id=virtio-disk5,bootindex=1 \ -L /usr/share/seabios How reproducible: Always reproducible Steps to Reproduce: 1. Start a Debian 8 guest with the parameters above 2. Start a Windows 2012 guest with the parameters above 3. Download iperf3 from http://www.iperf.fr for Both Windows and Linux 4. On the Linux guest, install the iperf3 package and run 'iperf3 -s' 5. On the Windows guest, install the iperf3 package and run 'iperf3 -c <iperf_server_address> -p 5201 6. Check the stats produced by iperf 7. Start another Debian 8 guest. Install iperf3 and run 'iperf3 -c <iperf_server_address> -p 5201 8. Compare stats with Windows guest Actual results: The Linux guest produces the following results: $ iperf3 -c 10.15.90.11 -p 5201 Connecting to host 10.15.90.11, port 5201 [ 4] local 10.15.90.12 port 49989 connected to 10.15.90.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 669 MBytes 5.61 Gbits/sec 0 3.23 MBytes [ 4] 1.00-2.00 sec 601 MBytes 5.04 Gbits/sec 0 3.23 MBytes [ 4] 2.00-3.00 sec 612 MBytes 5.14 Gbits/sec 0 3.23 MBytes [ 4] 3.00-4.00 sec 619 MBytes 5.19 Gbits/sec 0 3.23 MBytes [ 4] 4.00-5.00 sec 752 MBytes 6.31 Gbits/sec 0 3.23 MBytes [ 4] 5.00-6.00 sec 664 MBytes 5.57 Gbits/sec 0 3.23 MBytes [ 4] 6.00-7.00 sec 696 MBytes 5.84 Gbits/sec 0 3.23 MBytes [ 4] 7.00-8.00 sec 625 MBytes 5.24 Gbits/sec 0 3.23 MBytes [ 4] 8.00-9.00 sec 635 MBytes 5.32 Gbits/sec 0 3.23 MBytes [ 4] 9.00-10.00 sec 904 MBytes 7.59 Gbits/sec 0 3.23 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 6.62 GBytes 5.69 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 6.62 GBytes 5.69 Gbits/sec receiver On the other hand Windows guest produce only about 1.5 Gbps average bandwidth (see attached image). Expected results: Windows and Linux guests should show similar results. Additional info: Host lspci output: 00:00.0 Host bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (external gfx0 port A) (rev 02) 00:00.2 IOMMU: Advanced Micro Devices [AMD] nee ATI RD990 I/O Memory Management Unit (IOMMU) 00:02.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port B) 00:04.0 PCI bridge: Advanced Micro Devices [AMD] nee ATI RD890 PCI to PCI bridge (PCI express gpp port D) 00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] 00:12.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:12.1 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0 USB OHCI1 Controller 00:12.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:13.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller 00:13.1 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0 USB OHCI1 Controller 00:13.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller 00:14.0 SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 3d) 00:14.3 ISA bridge: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 LPC host controller 00:14.4 PCI bridge: Advanced Micro Devices [AMD] nee ATI SBx00 PCI to PCI Bridge 00:14.5 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI2 Controller 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:18.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:19.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:19.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:19.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:19.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:19.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:19.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1a.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1a.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1a.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1a.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1a.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1a.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1b.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1b.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1b.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1b.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1b.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1b.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1c.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1c.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1c.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1c.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1c.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1c.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1d.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1d.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1d.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1d.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1d.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1d.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1e.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1e.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1e.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1e.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1e.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1e.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 00:1f.0 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 0 00:1f.1 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 1 00:1f.2 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 2 00:1f.3 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 3 00:1f.4 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 4 00:1f.5 Host bridge: Advanced Micro Devices [AMD] Family 15h Processor Function 5 01:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a) 02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
Created attachment 1121169 [details] iperf stats for Windows 2012
Any progress on this?
Re-assigning wquan@ and yama@ to follow up this issue.
(In reply to Mohammed Gamal from comment #3) > Any progress on this? Just come back from spring festival holiday. And the machine is busy now, we will reproduce it once the machine is free.
QE CAN reproduce this issue with virtio-win-prewhql-112:win2012 and rhel7 guest show different results when do iperf,rhel7 guest 19Gbits/sec,win2012 4.97 Gbits/sec. rhel7 guest: # iperf3 -c 10.73.69.2 -p 5201 Connecting to host 10.73.69.2, port 5201 [ 4] local 10.73.68.159 port 53360 connected to 10.73.69.2 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.93 GBytes 16.6 Gbits/sec 0 2.88 MBytes [ 4] 1.00-2.00 sec 2.27 GBytes 19.5 Gbits/sec 0 3.07 MBytes [ 4] 2.00-3.00 sec 2.26 GBytes 19.5 Gbits/sec 0 3.07 MBytes [ 4] 3.00-4.00 sec 2.17 GBytes 18.6 Gbits/sec 0 3.07 MBytes [ 4] 4.00-5.00 sec 2.16 GBytes 18.6 Gbits/sec 0 3.07 MBytes [ 4] 5.00-6.00 sec 2.46 GBytes 21.1 Gbits/sec 0 3.07 MBytes [ 4] 6.00-7.00 sec 2.29 GBytes 19.7 Gbits/sec 0 3.07 MBytes [ 4] 7.00-8.00 sec 2.16 GBytes 18.5 Gbits/sec 0 3.07 MBytes [ 4] 8.00-9.00 sec 2.20 GBytes 18.9 Gbits/sec 0 3.07 MBytes [ 4] 9.00-10.00 sec 2.20 GBytes 18.9 Gbits/sec 0 3.07 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec receiver iperf Done. windows server2012 guest: Connecting to host 10.73.69.2, port 5201 [ 4] local 10.73.68.126 port 49203 connected to 10.73.69.2 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 591 MBytes 4.95 Gbits/sec [ 4] 1.00-2.00 sec 641 MBytes 5.38 Gbits/sec [ 4] 2.00-3.00 sec 624 MBytes 5.23 Gbits/sec [ 4] 3.00-4.00 sec 678 MBytes 5.69 Gbits/sec [ 4] 4.00-5.00 sec 616 MBytes 5.17 Gbits/sec [ 4] 5.00-6.00 sec 508 MBytes 4.27 Gbits/sec [ 4] 6.00-7.00 sec 580 MBytes 4.87 Gbits/sec [ 4] 7.00-8.00 sec 612 MBytes 5.14 Gbits/sec [ 4] 8.00-9.00 sec 532 MBytes 4.46 Gbits/sec [ 4] 9.00-10.00 sec 547 MBytes 4.59 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 5.79 GBytes 4.97 Gbits/sec sender [ 4] 0.00-10.00 sec 5.79 GBytes 4.97 Gbits/sec receiver iperf Done. package info: kernel-3.10.0-335.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64 virtio-win-prewhql-112 qemu cli: /usr/libexec/qemu-kvm -cpu SandyBridge -M pc -enable-kvm -m 2G -smp 2 -nodefconfig -nodefaults -rtc base=localtime,driftfix=slew -boot order=cd,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=112BLK201264KJ9,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga cirrus -qmp tcp:0:4444,server,nowait -monitor stdio -drive file=en_windows_server_2012_x64_dvd_915478.iso,if=none,id=drive-ide0-0-1,format=raw,cache=none,media=cdrom -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:3b:65:ee:ff According to performance team,windows guest performance is indeed slower than linux guest with netperf tool,let's wait for the detailed data.
(In reply to lijin from comment #6) > QE CAN reproduce this issue with virtio-win-prewhql-112:win2012 and rhel7 > guest show different results when do iperf,rhel7 guest 19Gbits/sec,win2012 > 4.97 Gbits/sec. > > rhel7 guest: > # iperf3 -c 10.73.69.2 -p 5201 > Connecting to host 10.73.69.2, port 5201 > [ 4] local 10.73.68.159 port 53360 connected to 10.73.69.2 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 1.93 GBytes 16.6 Gbits/sec 0 2.88 MBytes > > [ 4] 1.00-2.00 sec 2.27 GBytes 19.5 Gbits/sec 0 3.07 MBytes > > [ 4] 2.00-3.00 sec 2.26 GBytes 19.5 Gbits/sec 0 3.07 MBytes > > [ 4] 3.00-4.00 sec 2.17 GBytes 18.6 Gbits/sec 0 3.07 MBytes > > [ 4] 4.00-5.00 sec 2.16 GBytes 18.6 Gbits/sec 0 3.07 MBytes > > [ 4] 5.00-6.00 sec 2.46 GBytes 21.1 Gbits/sec 0 3.07 MBytes > > [ 4] 6.00-7.00 sec 2.29 GBytes 19.7 Gbits/sec 0 3.07 MBytes > > [ 4] 7.00-8.00 sec 2.16 GBytes 18.5 Gbits/sec 0 3.07 MBytes > > [ 4] 8.00-9.00 sec 2.20 GBytes 18.9 Gbits/sec 0 3.07 MBytes > > [ 4] 9.00-10.00 sec 2.20 GBytes 18.9 Gbits/sec 0 3.07 MBytes > > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec 0 sender > [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec > receiver > > iperf Done. > > windows server2012 guest: > Connecting to host 10.73.69.2, port 5201 > [ 4] local 10.73.68.126 port 49203 connected to 10.73.69.2 port 5201 > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-1.00 sec 591 MBytes 4.95 Gbits/sec > [ 4] 1.00-2.00 sec 641 MBytes 5.38 Gbits/sec > [ 4] 2.00-3.00 sec 624 MBytes 5.23 Gbits/sec > [ 4] 3.00-4.00 sec 678 MBytes 5.69 Gbits/sec > [ 4] 4.00-5.00 sec 616 MBytes 5.17 Gbits/sec > [ 4] 5.00-6.00 sec 508 MBytes 4.27 Gbits/sec > [ 4] 6.00-7.00 sec 580 MBytes 4.87 Gbits/sec > [ 4] 7.00-8.00 sec 612 MBytes 5.14 Gbits/sec > [ 4] 8.00-9.00 sec 532 MBytes 4.46 Gbits/sec > [ 4] 9.00-10.00 sec 547 MBytes 4.59 Gbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-10.00 sec 5.79 GBytes 4.97 Gbits/sec sender > [ 4] 0.00-10.00 sec 5.79 GBytes 4.97 Gbits/sec > receiver > > iperf Done. > > package info: > kernel-3.10.0-335.el7.x86_64 > qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64 > virtio-win-prewhql-112 > > qemu cli: > /usr/libexec/qemu-kvm -cpu SandyBridge -M pc -enable-kvm -m 2G -smp 2 > -nodefconfig -nodefaults -rtc base=localtime,driftfix=slew -boot > order=cd,menu=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=112BLK201264KJ9,if=none,id=drive-ide0-0-0,format=raw,serial=mike_cao, > cache=none -device > ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -chardev > pty,id=charserial0 -device isa-serial,chardev=charserial0,id=isa_serial0 > -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -vga cirrus -qmp > tcp:0:4444,server,nowait -monitor stdio -drive > file=en_windows_server_2012_x64_dvd_915478.iso,if=none,id=drive-ide0-0-1, > format=raw,cache=none,media=cdrom -device > ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev > tap,script=/etc/qemu-ifup,downscript=no,id=hostnet0,vhost=on -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=00:52:3b:65:ee:ff > > According to performance team,windows guest performance is indeed slower > than linux guest with netperf tool,let's wait for the detailed data. Are those guest to guest results? Both sides are Windows and both sides are RHEL? Windows ha deficiency without enabled RSC on receive side and will definitely be slower than Linux. While we have implemented RSC - it can be enabled only in compile time because we cannot yet certify such a driver. Can you test Windows as iperf client (TX only)?
Any progress with this?
Test 10 virtio-win-prewhql builds, it's not a regression issue. is an old issue from following results. rhel 7 guest as iperf3 receiver (iper3 -s) win2012 64 bits as iperf3 sender (iper3 -c $rhel7_guest_ip -p 5201) package info: host/guest 3.10.0-327.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.7.x86_64 +---------------+---------------+---------------+---------------+---------------+ | rhel7 guest |2012-build112 |2012-build109 |2012-build106 |2012-build103 | +---------------+---------------+---------------+---------------+---------------+ | 17.4 Gbits/sec|8.92 Gbits/sec | 8.84 Gbits/sec| 8.76 Gbits/sec| 8.87 Gbits/sec| +---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+ |2012-build100 |2012-build97 |2012-build94 | 2012-build91 | +---------------+---------------+---------------+---------------+ |8.84 Gbits/sec | 9.07 Gbits/sec| 9.07 Gbits/sec| 8.91 Gbits/sec| +---------------+---------------+---------------+---------------+
(In reply to Mohammed Gamal from comment #11) > Any progress with this? please look at comment12.
(In reply to Yanhui Ma from comment #13) > (In reply to Mohammed Gamal from comment #11) > > Any progress with this? > > please look at comment12. Does this mean it's a known issue? Is there an intention to fix it?
(In reply to Mohammed Gamal from comment #14) > (In reply to Yanhui Ma from comment #13) > > (In reply to Mohammed Gamal from comment #11) > > > Any progress with this? > > > > please look at comment12. > > Does this mean it's a known issue? Is there an intention to fix it? Sure, we will look into performance issues for next release.
Any updates on this?
(In reply to Mohammed Gamal from comment #16) > Any updates on this? It is in our development queue.
The receive side performance is related to RSC feature that is currently disabled in our driver (see BZ #950611). We are working to have certified driver with this feature turned on.
What's the latest state of this bug? I am not sure exactly what is the state of BZ #950611
(In reply to Mohammed Gamal from comment #25) > What's the latest state of this bug? I am not sure exactly what is the state > of BZ #950611 In progress. Patches will be pushed to upstream soon.
*** This bug has been marked as a duplicate of bug 950611 ***