Bug 969791

Amir,

This was filed under RHEV-M, just want to make sure it's in the right location. 
Is this just a libvirt bug or is it occurring under RHEV-H/RHEV-M controlled machines?

Seems like libvirtd bug only.
Thanks.

Did these steps try to reproduce this issue:

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6.x86_64
kernel:
2.6.32-276.el6.x86_64

Steps to Reproduce:
1. check the eth0 is not connected to bridge 
# brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		8000.5254000cba2d	yes		virbr0-nic

2. # ethtool -k eth0|grep large-receive-offload
large-receive-offload: off

3. # service libvirtd status
libvirtd is stopped

4. # ethtool -K eth0 lro on
Cannot set large receive offload settings: Operation not supported

5. # ethtool -i eth0
driver: tg3
version: 3.122
firmware-version: 5761-v3.77
bus-info: 0000:3f:00.0

Actual results:
In step4: failed to set lro on:"Operation not supported", so failed to reproduce this issue.

There is a related bug:https://bugzilla.redhat.com/show_bug.cgi?id=818504 
Bug 818504 - Disable LRO for all NICs that have LRO enabled [rhel-6.2.z]

Hi, amira

Could you please provide the kernel version and the interface's driver type ?

Kernel is 2.6.32.279
Driver is mlx4_en (Mellanox)
You should be able to reproduce with ixgbe / bnx2 / be2net as well.

Did these steps to reproduce this issue:

Version-Release number of selected component (if applicable):
libvirt-0.9.10-21.el6.x86_64
kernel:
2.6.32-279.el6.x86_64

Steps to Reproduce:

1. check the eth0 is connected to the brige.
# brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.3440b59f1390	no		eth0
 
2. check the bond0 is not connected to the brige, and with lro off
# modprobe bonding mode=1 miimon=100
# ifconfig bond0 up
# ifenslave bond0 eth0
# ethtool -k bond0|grep large
large-receive-offload: off

3. check the libvirtd is stopped 
# service libvirtd status
libvirtd is stopped

4. set the bond0 lro on 
# ethtool -K bond0 lro on
# ethtool -k bond0|grep large
large-receive-offload: on

5. start libvirtd 
# service libvirtd start
Starting libvirtd daemon:        [ OK ]

6. check the bond0 lro off 
#ethtool -k bond0|grep large
large-receive-offload: off

7. # ethtool -i eth0
driver: bnx2
version: 2.2.1
firmware-version: bc 6.2.0 NCSI 2.0.11
bus-info: 0000:0b:00.0

(In reply to chhu from comment #6)
> Did these steps to reproduce this issue:
> 
> Version-Release number of selected component (if applicable):
> libvirt-0.9.10-21.el6.x86_64
> kernel:
> 2.6.32-279.el6.x86_64
> 
> Steps to Reproduce:
> 
> 1. check the eth0 is connected to the brige.
> # brctl show
> bridge name	bridge id		STP enabled	interfaces
> switch		8000.3440b59f1390	no		eth0
>  
> 2. check the bond0 is not connected to the brige, and with lro off
> # modprobe bonding mode=1 miimon=100
> # ifconfig bond0 up
> # ifenslave bond0 eth0
> # ethtool -k bond0|grep large
> large-receive-offload: off
> 
> 3. check the libvirtd is stopped 
> # service libvirtd status
> libvirtd is stopped
> 
> 4. set the bond0 lro on 
> # ethtool -K bond0 lro on
> # ethtool -k bond0|grep large
> large-receive-offload: on
> 
> 5. start libvirtd 
> # service libvirtd start
> Starting libvirtd daemon:        [ OK ]
> 
> 6. check the bond0 lro off 
> #ethtool -k bond0|grep large
> large-receive-offload: off
> 
> 7. # ethtool -i eth0
> driver: bnx2
> version: 2.2.1
> firmware-version: bc 6.2.0 NCSI 2.0.11
> bus-info: 0000:0b:00.0

Actual results:
When libvirtd starts it enables the global ip forward sysctl which then turns lro off for all interfaces.

Expected results:
libvirtd will not affect unconnected interface offloads.

Any due date for a solution ?

Since libvirt is also using iptables rules for routing (masquerading) of VM traffic into the internet, forwarding needs to be enabled globally when used with the default libvirt network. It's impossible to determine which interfaces need to be enabled for forwarding to work in various configurations and thus libvirt enables global forwarding.

For the configuration described above there are two possible workarounds:

1) Don't use libvirt managed networks
When no libvirt network is used the forwarding option isn't turned on. The user is then able to fine tune network configuration and forwarding on a per interface basis. The guests then may be bridged using the following config:

    <interface type='bridge'>
      <mac address='52:54:00:21:e4:9a'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2) Manually disable forwarding on interfaces that need LRO
The second option is to start libvirt with the network and then manually tweak the interface settings to disable forwarding for the specified interfaces that don't require it and enable LRO and other tunables.

I recommend closing this bug as there isn't a straightforward solution for this problem and workarounds exist. Also the requested use case isn't common and thus covered by the default libvirt network behavior.

Closing as CANTFIX due to the reasons described above.

Peter, could you elaborate on "It's impossible to determine which interfaces need to be enabled for forwarding"? Why should a non-NATed layer-2-only libvirt network should be affected by IP forwarding on another network?

(In reply to Dan Kenigsberg from comment #11)
> Peter, could you elaborate on "It's impossible to determine which interfaces
> need to be enabled for forwarding"? Why should a non-NATed layer-2-only
> libvirt network should be affected by IP forwarding on another network?

When you enable IPv4 IP forwarding in the kernel globally (echo 1 > /proc/sys/net/ipv4/ip_forward) this enables IP forwarding on ALL interfaces and the kernel then disables LRO for those interfaces. IP forwarding can be enabled on a per-interface basis, but it's impossible to know for libvirt which interfaces this needs to be done on as there might be multiple routes into multiple networks that the user might want to have acccess to.

As the configuration requested originally is really advanced, the user still has the option to configure the network manually. With the knowledge of the network topology, the right decision can be made and IP forwarding can be enabled only on the interfaces that are required to relay traffic.