Bug 855640

Summary: Unusable very slow inbound network to macvtap VMs
Product: [Fedora] Fedora Reporter: Richard Davies <richard>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 19CC: amit.shah, arnold.x.wang, berrange, brian, cfergeau, crobinso, dwmw2, igeorgex, itamar, javier.wilson, joanayma, knoel, pbonzini, rjones, scottt.tw, stik3mup, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-11 19:19:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Richard Davies 2012-09-09 14:06:51 UTC
Description of problem:

Inbound data transfers to a Fedora 17 qemu-kvm VM on a Fedora 17 virtualization host from a different physical machine over local gigabit ethernet start reasonably fast, but slow down to very slow speeds (<100KB/s) within 30 seconds of starting.


Version-Release number of selected component (if applicable):

Fedora 17 current live-cd
kernel 3.3.4-5.fc17.x86_64
qemu-kvm-1.0.1-1.fc17.x86_64


How reproducible:

The problem happens for every VM that I start


Steps to Reproduce:
1. Have two physical machines on the same local gigabit ethernet network. The tester machine and the virtualization host.
2. Install Fedora 17, qemu-kvm and virt-manager on the virtualization host
3. Start a new virtual machine in Virtual Machine Manager, specifying the virtual network as "Host device p3p1: macvtap" and "Source mode: Bridge"
4. Test a large inbound data transfer (e.g. a large scp) into the VM from the other physical machine over the local gigabit ethernet network.

[I include a complete list of steps to duplicate using only a Fedora 17 live CD below]


Actual results:

Inbound data transfer starts reasonably fast, but rapidly slows to <100KB/s.


Expected results:

Inbound data transfer should be similar speed to the VM as to the virtualization host.


Additional info:

Here are the complete list of steps to duplicate using only a Fedora 17 live CD:

0. Assumes a virtualization host and a second physical test machine connected on a gigabit ethernet network with a DHCP server allocating IPs.

1. Boot the Fedora 17 live CD on the virtualization host and click on "Try Fedora"

2. At a root prompt run:

yum install virt-manager libvirt qemu-kvm gtk-vnc-devel openssh-server
service libvirtd start
service sshd start
iptables -F
passwd

3. Run virt-manager and create a new virtual machine:

Name: test
Local install media
Use CDROM or DVD: Fedora-17-x86_64-Live-Desktop.iso (/dev/sr0)
OS type: Linux
Version: Fedora 17
Memory: 1024MB
CPUs: 1
Disable storage for this virtual machine (will use Live CD only)
Checked: Customize before install
Advanced options: no changes

In the settings, click on NIC, select "Host device p3p1: macvtap" with "Source mode: Bridge" and click "Apply"

Click "Begin installation"

4. The Fedora 17 live CD will boot in the VM. Click on "Try Fedora"

5. At a root prompt inside the VM, run:

yum install openssh-server
service sshd start
iptables -F
passwd

6. From the external physical test machine test network transfer speed to and from the virtualization host and to and from the VM (e.g. with scp).

You will find that the following commands are fast run on the test machine:

scp ~/1GB-file root@virtualization-host:1GB-file
scp root@virtualization-host:1GB-file ~/1GB-file
scp root@vm:1GB-file ~/1GB-file

But the following command run on the test machine starts fast but rapidly slows down to <100KB/s:

scp ~/1GB-file root@vm:1GB-file

The same holds for other large network transfers - I am simply testing with scp since it gives a visible speed indication.

Comment 1 Richard Davies 2012-09-09 14:10:43 UTC
Bug 853669 may be related. I have written a new report since I can give full reproduction instructions using only Fedora 17 for the virtualization host and the VM - no Windows 7 is required.

Comment 2 Richard Davies 2012-09-09 14:12:46 UTC
I believe this is a bug in the macvtap kernel component, as discussed but not yet solved in this thread: http://marc.info/?t=134511098600002

Comment 3 Richard Davies 2012-09-09 16:47:04 UTC
Note: For some reason, this bug shows up most clearly on faster networks - i.e. it replicates best on a wired gigabit ethernet network.

On slower networks like wifi, the bug is less easily distinguished and also seemingly less severe(!)

Comment 4 Paolo Bonzini 2012-09-11 11:02:58 UTC
*** Bug 853669 has been marked as a duplicate of this bug. ***

Comment 5 javierwilson 2013-04-05 18:47:46 UTC
I can confirm. Speed slows down to 5KB/s
I am using macvtap and have a gigabit ethernet network.

Comment 6 javierwilson 2013-04-16 12:29:22 UTC
I solved the problem switching from "default" NIC to "virtio" which supports gigabit speeds.
I just wanted to comment that my host machine and my switch were both using gigabit speeds.

Comment 7 Fedora End Of Life 2013-07-04 04:13:56 UTC
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 8 Cole Robinson 2013-07-11 19:19:48 UTC
I'm pretty sure this isn't an issue with latest kernels, but someone reopen if they can reproduce on F19.

Comment 9 Joan AymĂ  2013-08-05 14:54:20 UTC
Can confirm this bug on Fedora 19 on latest kernel 3.9.5-301.fc19.x86_64, virt-manager-0.10.0-1, qemu-1.4.2-4.

CPU: Intel Core i7. 
NIC: Realtek RTL8168evl.

Tried with virtio and others with same result.
The discussion on this thread http://marc.info/?t=134511098600002 suggests that's a CONFIG_INTEL_IDLE but also bug is present on AMD also.

Comment 10 Shane Ward 2013-08-16 15:17:44 UTC
I can also confirm this bug on a Fedora 19 setup.

kernel - 3.10.6-200.fc19.x86_64
qemu   - 1.4.2-5

Having the problem across multiple Dell PowerEdge servers (1950, R610, R710, R810) all with Intel Xenon processors and various Broadcom NIC models.

Outbound speeds are normal, but anything inbound extremely slow.

Comment 11 Brian Pitts 2013-10-22 18:01:19 UTC
I had these same symptoms on a RHEL6 host with a RHEL6 guest. Turning off generic receive offload on the NIC I was using for macvtap solved the problem for me; e.g. ethtool -K eth0 gro off

This workaround is described at https://access.redhat.com/site/solutions/349403