Bug 474191

Are both VMs paravirt and running the same kernel version as dom0? Is your TCP traffic IPv4 (we only support IPv4 virtual forwarding in Xen in RHEL5).

Also what processing does VM1 do to the packets, e.g., please list netfilter rules and other relevant configuration details in VM1.

Please also test if disabling tx checksums in VM1 fixes the problem.  If that doesn't, please test disabling tx checksums in dom0's netback interface to VM1.

Thanks!

Both VMs are paravirt and runnning the same kernel as dom0.

All the trafic is IPv4.

The netfilter rules involved filtering (iptables -t filter), marking (mangle -t mangle), natting (-t nat).

If I remove all rules but the nat one :

[root@sol ~]# iptables -L -n -v -t nat
Chain PREROUTING (policy ACCEPT 11M packets, 1925M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 5296K packets, 381M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  247 17199 SNAT       all  --  *      eth3    192.168.240.0/24     0.0.0.0/0           to:195.189.65.250 

Chain OUTPUT (policy ACCEPT 5257K packets, 378M bytes)
 pkts bytes target     prot opt in     out     source               destination         

The result is the same (my intranet host is on net 192.168.240.0/24, the VM1 has address 195.189.65.250 and the VM2 has address 195.189.65.253).


If I remove this last one (so that their is no iptables rules at all), the problem disapear.

So the problem is related to tx offload AND iptables nat. A simple solution is to issue "ethtool -K eth0 tx off" in the VM2.

If I issue a "ethtool -K eth3 tx off" in the VM1, the problem is still present.

If I issue a "ethtool -K vif4.3 tx off" on the physical host to disable the tx checksum in dom0's netback interface to VM1, the problem disapear.

Regards,

Right, unfortunately we can't support NAT + TX offload in RHEL5 for packets originating from another guest because the underlying netfilter infrastructure for doing so simply isn't there.  Disabling checksum offload on the interface going into VM1 (vif4.3) is the best workaround.

Is there any other bug tracking number int netfilter that we can follow in order to keep an eye on the resolution of this problem ?

The vif interface (here vif4.3) is created during the xen vm launch. In my case, this is in fact launched by cman's rgmanager on one of the physical servers of the cluster. 

That is the best way to intercept the vm creation on the physical host in order to issue the required "ethtool -K vif4.3 tx off" ? 

Is there any hook in the xen package where we can for example interpret a vif = [ "mac=00:16:3e:72:4e:d2,bridge=vlan50,txoffload=off" ] ?

Waiting for such a solution, the best way for me was to disabled it in the VM2, so that it works with rgmanager.

Alain, IIRC /etc/xen/scripts/vif-bridge is the one you can use to disable TX offload on vif4.3.

Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Due to technical problems with passing TX checksum offload information between paravirtual domains, we do not support the use of TX checksum offload in conjunction with NAT for traffic originating from another domain.  In other words, you may use TX checksum offload together with NAT as long as the NAT rule is applied in the domain where the traffic originates.

 If you have to apply NAT to traffic originating from another domain, then TX checksum should be disabled on the vif interface entering the domain with the rules.

 Note that this also applies to fully virtualised domains using paravirtual network drivers.  Fully virtualised domains using fully virtualised drivers are not affected as they do not support TX checksum offload at all.

Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,5 +1,3 @@
-Due to technical problems with passing TX checksum offload information between paravirtual domains, we do not support the use of TX checksum offload in conjunction with NAT for traffic originating from another domain.  In other words, you may use TX checksum offload together with NAT as long as the NAT rule is applied in the domain where the traffic originates.
+Due to technical problems with passing TX checksum offload information between paravirtual domains, the use of TX checksum offload in conjunction with NAT for traffic originating from another domain is not supported. TX checksum offload can be used together with NAT as long as the NAT rule is applied in the domain where the traffic originates.
 
- If you have to apply NAT to traffic originating from another domain, then TX checksum should be disabled on the vif interface entering the domain with the rules.
+Note that this also applies to fully virtualised domains using paravirtual network drivers. Fully virtualised domains using fully virtualised drivers are not affected as they do not support TX checksum offload at all.-
- Note that this also applies to fully virtualised domains using paravirtual network drivers.  Fully virtualised domains using fully virtualised drivers are not affected as they do not support TX checksum offload at all.

The following is perhaps a duplication of the information above but it would have helped me, and hopefully others, isolate the problem and provide a solution. Our setup is fairly convoluted with load balanced, connection tracking and traffic marking and shaping so we stumbled on to this bug report almost by chance.

We experience the same problem with Dom0 when using PREROUTING nat rules. The problems are however not limited to communications to/from DomU guests.

Dom0 running Squid proxy server (transparent port configured on tcp:3129)
iptables -t nat -A PREROUTING -s 192.168.0.0/16 -d ! 192.168.1.0/29 -p tcp --dport 80 -j DNAT --to 192.168.1.1:3129

Clients are able to retrieve about 100KB without problems but connection thereafter gets extremely slow and then eventually stops altogether. If we however skip NAT and get workstations to configure their browsers to connect directly to the Squid proxy server everything works perfectly well.

Running Xen with a single domU (Windows 2003 with Xen paravirtualisation drivers).

Dom0 eth0: 192.168.1.1/29
DomU NIC1: 192.168.1.3/29

PS: Dom0 has routes for 192.168.0.0/16 to 192.168.1.2 (a WiFi router).

eth0 - xenbr0


Hardware: Confirmed to affect Intel Original S3210SH and S5000VSA motherboards
with Intel 80003ES2LAN, 82566DM-2 and 82541GI embedded Gigabit Ethernet Controllers.

ethtool -k eth0
  Offload parameters for eth0:
  Cannot get device rx csum settings: Operation not supported
  Cannot get device udp large send offload settings: Operation not supported
  rx-checksumming: off
  tx-checksumming: on
  scatter-gather: off
  tcp segmentation offload: off
  udp fragmentation offload: off
  generic segmentation offload: off

Adding 'ethtool -K eth0 tx off' to /etc/rc.d/rc.local fixes the problem...

This issue cannot be fixed and has been release noted. Closing. If the workaround is not sufficient, please reopen.