Bug 969791 - libvirtd service enable global ip forwarding instead of just the bridge interface causing low throughput
libvirtd service enable global ip forwarding instead of just the bridge inter...
Status: CLOSED CANTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.6
Unspecified Linux
unspecified Severity high
: rc
: ---
Assigned To: Peter Krempa
Virtualization Bugs
network
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-02 07:42 EDT by amira
Modified: 2013-07-11 03:57 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-10 10:47:32 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description amira 2013-06-02 07:42:06 EDT
Description of problem:
When libvirtd starts it enables the global ip forward sysctl which then turns lro off for all interfaces. it can instead enable just the bridge (br/vibr) and disable ip forwarding only for interfaces which are connected to the bridge. 

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1.service libvirtd stop
2.ethtool -K eth4 lro on (eth4 is not connected to bridge)
3.service libvirtd start 
4.ethtool -k eth4 ( here lro is off)

Actual results:


Expected results:
libvirtd will not affect unconnected interface offloads.

Additional info:
Comment 1 Joey Boggs 2013-06-03 09:23:03 EDT
Amir,

This was filed under RHEV-M, just want to make sure it's in the right location. 
Is this just a libvirt bug or is it occurring under RHEV-H/RHEV-M controlled machines?
Comment 2 amira 2013-06-05 03:09:09 EDT
Seems like libvirtd bug only.
Thanks.
Comment 4 chhu 2013-06-12 23:11:03 EDT
Did these steps try to reproduce this issue:

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6.x86_64
kernel:
2.6.32-276.el6.x86_64

Steps to Reproduce:
1. check the eth0 is not connected to bridge 
# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.5254000cba2d	yes		virbr0-nic

2. # ethtool -k eth0|grep large-receive-offload
large-receive-offload: off

3. # service libvirtd status
libvirtd is stopped

4. # ethtool -K eth0 lro on
Cannot set large receive offload settings: Operation not supported

5. # ethtool -i eth0
driver: tg3
version: 3.122
firmware-version: 5761-v3.77
bus-info: 0000:3f:00.0

Actual results:
In step4: failed to set lro on:"Operation not supported", so failed to reproduce this issue.

There is a related bug:https://bugzilla.redhat.com/show_bug.cgi?id=818504 
Bug 818504 - Disable LRO for all NICs that have LRO enabled [rhel-6.2.z]

Hi, amira

Could you please provide the kernel version and the interface's driver type ?
Comment 5 amira 2013-06-13 04:27:28 EDT
Kernel is 2.6.32.279
Driver is mlx4_en (Mellanox)
You should be able to reproduce with ixgbe / bnx2 / be2net as well.
Comment 6 chhu 2013-06-14 01:59:00 EDT
Did these steps to reproduce this issue:

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6.x86_64
kernel:
2.6.32-279.el6.x86_64

Steps to Reproduce:

1. check the eth0 is connected to the brige.
# brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.3440b59f1390	no		eth0
 
2. check the bond0 is not connected to the brige, and with lro off
# modprobe bonding mode=1 miimon=100
# ifconfig bond0 up
# ifenslave bond0 eth0
# ethtool -k bond0|grep large
large-receive-offload: off

3. check the libvirtd is stopped 
# service libvirtd status
libvirtd is stopped

4. set the bond0 lro on 
# ethtool -K bond0 lro on
# ethtool -k bond0|grep large
large-receive-offload: on

5. start libvirtd 
# service libvirtd start
Starting libvirtd daemon:        [ OK ]

6. check the bond0 lro off 
#ethtool -k bond0|grep large
large-receive-offload: off

7. # ethtool -i eth0
driver: bnx2
version: 2.2.1
firmware-version: bc 6.2.0 NCSI 2.0.11
bus-info: 0000:0b:00.0
Comment 7 chhu 2013-06-14 02:04:14 EDT
(In reply to chhu from comment #6)
> Did these steps to reproduce this issue:
> 
> Version-Release number of selected component (if applicable):
> libvirt-0.9.10-21.el6.x86_64
> kernel:
> 2.6.32-279.el6.x86_64
> 
> Steps to Reproduce:
> 
> 1. check the eth0 is connected to the brige.
> # brctl show
> bridge name	bridge id		STP enabled	interfaces
> switch		8000.3440b59f1390	no		eth0
>  
> 2. check the bond0 is not connected to the brige, and with lro off
> # modprobe bonding mode=1 miimon=100
> # ifconfig bond0 up
> # ifenslave bond0 eth0
> # ethtool -k bond0|grep large
> large-receive-offload: off
> 
> 3. check the libvirtd is stopped 
> # service libvirtd status
> libvirtd is stopped
> 
> 4. set the bond0 lro on 
> # ethtool -K bond0 lro on
> # ethtool -k bond0|grep large
> large-receive-offload: on
> 
> 5. start libvirtd 
> # service libvirtd start
> Starting libvirtd daemon:        [ OK ]
> 
> 6. check the bond0 lro off 
> #ethtool -k bond0|grep large
> large-receive-offload: off
> 
> 7. # ethtool -i eth0
> driver: bnx2
> version: 2.2.1
> firmware-version: bc 6.2.0 NCSI 2.0.11
> bus-info: 0000:0b:00.0

Actual results:
When libvirtd starts it enables the global ip forward sysctl which then turns lro off for all interfaces.

Expected results:
libvirtd will not affect unconnected interface offloads.
Comment 8 amira 2013-06-16 03:43:56 EDT
Any due date for a solution ?
Comment 9 Peter Krempa 2013-07-02 11:05:00 EDT
Since libvirt is also using iptables rules for routing (masquerading) of VM traffic into the internet, forwarding needs to be enabled globally when used with the default libvirt network. It's impossible to determine which interfaces need to be enabled for forwarding to work in various configurations and thus libvirt enables global forwarding.

For the configuration described above there are two possible workarounds:

1) Don't use libvirt managed networks
When no libvirt network is used the forwarding option isn't turned on. The user is then able to fine tune network configuration and forwarding on a per interface basis. The guests then may be bridged using the following config:

    <interface type='bridge'>
      <mac address='52:54:00:21:e4:9a'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2) Manually disable forwarding on interfaces that need LRO
The second option is to start libvirt with the network and then manually tweak the interface settings to disable forwarding for the specified interfaces that don't require it and enable LRO and other tunables.

I recommend closing this bug as there isn't a straightforward solution for this problem and workarounds exist. Also the requested use case isn't common and thus covered by the default libvirt network behavior.
Comment 10 Peter Krempa 2013-07-10 10:47:32 EDT
Closing as CANTFIX due to the reasons described above.
Comment 11 Dan Kenigsberg 2013-07-10 14:36:26 EDT
Peter, could you elaborate on "It's impossible to determine which interfaces need to be enabled for forwarding"? Why should a non-NATed layer-2-only libvirt network should be affected by IP forwarding on another network?
Comment 12 Peter Krempa 2013-07-11 03:57:16 EDT
(In reply to Dan Kenigsberg from comment #11)
> Peter, could you elaborate on "It's impossible to determine which interfaces
> need to be enabled for forwarding"? Why should a non-NATed layer-2-only
> libvirt network should be affected by IP forwarding on another network?

When you enable IPv4 IP forwarding in the kernel globally (echo 1 > /proc/sys/net/ipv4/ip_forward) this enables IP forwarding on ALL interfaces and the kernel then disables LRO for those interfaces. IP forwarding can be enabled on a per-interface basis, but it's impossible to know for libvirt which interfaces this needs to be done on as there might be multiple routes into multiple networks that the user might want to have acccess to.

As the configuration requested originally is really advanced, the user still has the option to configure the network manually. With the knowledge of the network topology, the right decision can be made and IP forwarding can be enabled only on the interfaces that are required to relay traffic.

Note You need to log in before you can comment on or make changes to this bug.