Bug 438227

I wonder if the igb driver has the same packet split issue we saw in e1000e....

/me goes to check

There could be a problem since igb disables hardware crc stripping but doesn't
ever decrement the len of the frame.  Maybe not though.  If we actually had some
igb hardware that we could use for testing you can probably reproduce this issue
by just putting igb in a bridge and making the bridge the endpoint for traffic
-- but I'm sure Herbert knows that. :-)

Is tx offload enabled on the tg3? Do we see this with domU to domU? Thanks1

On the system with the issue the Dom0 is running an igb driver with tx offload
enabled by default
[root@perf10 np2.4]# ethtool -i peth1
driver: igb
version: 1.0.1
firmware-version: 1.11-5
bus-info: 0000:08:00.1
[root@perf10 np2.4]# ethtool -k peth1
Offload parameters for peth1:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
[root@perf10 np2.4]# ethtool -k eth1
Offload parameters for eth1:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off


On the working system (with the tg3 driver)[root@specclient3 ~]# ethtool -i peth1
driver: tg3
version: 3.86
firmware-version: 5715-v3.28
bus-info: 0000:09:04.1
[root@specclient3 ~]# ethtool -k peth1
Offload parameters for peth1:
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off
[root@specclient3 ~]# ethtool -k eth1
Offload parameters for eth1:
Cannot get device rx csum settings: Operation not supported
Cannot get device udp large send offload settings: Operation not supported
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: off


------------------

Two rhel51_pv guests on the same Dom0 can communicate "just fine" on the system
with the igb driver. 



---------------

At this point in time, the issue is with an external system trying to run the
tests against the guests.  A quick matrix (where ext stands for an external box)

domU <-> domU      Pass
dom0 <-> domU      Pass
ext  <-> dom)      Pass
ext  <-> PV guest  Fails
ext  <-> FV Guest  Pass

By default the FV guest has the tx offload disabled, enabling it does not seem
to have a negative impact.   

Turning off the tx offload on the PV guest allows things to work.

My testing indicates a problem with the driver.  dom0 doesn't show it presumably
because of its NAT netfilter rule which causes the checksum to be computed in
software.

I've takled to Auke from Intel and he says he has some patches which may help. 
I'll check with him again tomorrow.

I was told by Auke that this patch should help:

commit 44b0cda37534093fd9fefacd64d5fbb589c50795
Author: Mitch Williams <mitch.a.williams>
Date:   Fri Mar 7 10:32:13 2008 -0800

    igb: Correctly get protocol information

    We can't look at the socket to get protocol information. We should
    instead look directly at the packet, and hope there are no IPv6
    option headers.

    Signed-off-by: Mitch Williams <mitch.a.williams>
    Signed-off-by: Auke Kok <auke-jan.h.kok>
    Signed-off-by: Jeff Garzik <jeff>

Please let me know if it doesn't and I'll ask him for more :)

Um, sorry but I have no idea what to do with a commit number at this point. 

some additional pointers / direction are needed. 

The fix list in comment #6 was based on a patch upstream in February that I
guess morphed into the fix that was finally committed in March.

http://marc.info/?l=linux-netdev&m=120278022027115&w=2

We can take the upstream fix without too much trouble, but my original fix was
designed to prevent a panic when skb->sk was NULL since traffic on the bridge
wasn't sent on a socket.

It looks like the only thing this change actually does functionally as compared
to my patch is add this flag E1000_ADVTXD_TUCMD_L4T_TCP if the traffic is
ipv6/tcp traffic.

Either patch should do the trick.  The problem is that the driver is currently
looking up the socket to get the protocol.  But of course for Xen there is no
socket since the packet is from the guest.

Created attachment 300247 [details]
igb: Correctly get protocol information

Here is the patch that should fix the problem.

My test kernels have been updated to include a patch for this bugzilla.

http://people.redhat.com/agospoda/#rhel5

Please test them and report back your results.

I used Andy's test kernel 2.6.18-89.el5.gtest.46xen and tested with a rhel51 PV
guest. I ran some 60 sec runs of netperf using both tcp and udp (ipv4 only).  I
did not observe the previously mentioned issues.

I retested the original issue reported by rebooting the host with the stock
2.6.18-88 kernel and retesting.  In this situation the problem did occur as the
driver did not have the patch.  

Looking back at the patch I see that the original one did have a NULL pointer
check, so I understand why this would help (of course know it does helps too!)
and not just prevent a panic.  Score another one for Herbert. :-)

Updating PM score.