Bug 582367 - implement dev_disable_lro for RHEL5
Summary: implement dev_disable_lro for RHEL5
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.6
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Neil Horman
QA Contact: Hangbin Liu
URL:
Whiteboard:
Keywords: ZStream
: 518531 586352 (view as bug list)
Depends On:
Blocks: 514491 584359 596385 612856
TreeView+ depends on / blocked
 
Reported: 2010-04-14 18:26 UTC by Neil Horman
Modified: 2018-10-27 12:03 UTC (History)
15 users (show)

(edit)
Large-Receive Offload (LRO) is a performance optimization that enables the kernel to fetch and process, as a unit, more than one received packet from a network device. It was previously not possible to dynamically disable LRO for devices in a forwarding mode. This has been fixed with this update so that the kernel is able to dynamically disable LRO for devices in a forwarding state, or which had had LRO turned on manually.
Clone Of:
: 584359 (view as bug list)
(edit)
Last Closed: 2011-01-13 21:27:23 UTC


Attachments (Terms of Use)
omnibus patch for this bug (13.40 KB, patch)
2010-04-14 20:22 UTC, Neil Horman
no flags Details | Diff
infrastrucuture dev_disable_lro patch (6.98 KB, patch)
2010-04-15 19:44 UTC, Neil Horman
no flags Details | Diff
bnx2x patch (1.45 KB, patch)
2010-04-15 19:45 UTC, Neil Horman
no flags Details | Diff
enic patch (1.54 KB, patch)
2010-04-15 19:45 UTC, Neil Horman
no flags Details | Diff
mlx4 patch (1.97 KB, patch)
2010-04-15 19:46 UTC, Neil Horman
no flags Details | Diff
s2io patch (2.12 KB, patch)
2010-04-15 19:46 UTC, Neil Horman
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0017 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 5.6 kernel security and bug fix update 2011-01-13 10:37:42 UTC

Internal Trackers: 786403

Description Neil Horman 2010-04-14 18:26:28 UTC
Description of problem:
Nominally LRO and devices in a forwarding mode are mutually exclusive.  Upstream has the ability to dynamically disable LRO for devices put into a forwarding state, while RHEL5 does not.  The current RHEL5 solution is for drivers supporting LRO, we default disable it using the driver specific module options, but this doesn't cover the case where users enable LRO specifically. in these cases forwarding can result in various odd behavior if LRO is on.

Version-Release number of selected component (if applicable):



Additional info:

curently these driver require the lro_disable feature:
s2io
mlx4_en
bnx2x
enic

Comment 1 Neil Horman 2010-04-14 20:22:37 UTC
Created attachment 406634 [details]
omnibus patch for this bug

This is my first pass at this patch.  It should be separated out before posting to a core and individual driver components.

I'm validating it on bnx2x currently.  Need to find mlx4, enic and s2io cards to test there prior to posting.

Comment 2 Neil Horman 2010-04-14 20:36:57 UTC
note, need to validate that the NIC is up prior to resetting the hardware, or we get wierd behavior.  Just got a BUG on bnx2x in testing, which resulted from adding an interface to a bridge while the interface was down. This will likely be the case for all affected drivers, as their lro configuration occurs during probe on module insert, rather than on device open/close.

Comment 3 Neil Horman 2010-04-15 19:44:14 UTC
Created attachment 406909 [details]
infrastrucuture dev_disable_lro patch

This is the cleaned up broken out patch series for this bug

Comment 4 Neil Horman 2010-04-15 19:45:08 UTC
Created attachment 406910 [details]
bnx2x patch

Comment 5 Neil Horman 2010-04-15 19:45:45 UTC
Created attachment 406911 [details]
enic patch

Comment 6 Neil Horman 2010-04-15 19:46:41 UTC
Created attachment 406912 [details]
mlx4 patch

Comment 7 Neil Horman 2010-04-15 19:46:58 UTC
Created attachment 406913 [details]
s2io patch

Comment 8 Stanislaw Gruszka 2010-04-16 14:30:04 UTC
*** Bug 518531 has been marked as a duplicate of this bug. ***

Comment 11 Siert Z. 2010-05-13 11:25:03 UTC
I tested Stanislaw's xen kernel on RHEL5.5 x86_64 (http://people.redhat.com/sgruszka/rhel5/bz573114/). 

The hardware: HP BL460c + bnx2x (Virtual connect - Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe).

[root@hsl0000 ~]# uname -a
Linux hsl0000.domain.local 2.6.18-197.el5.bnx2x_testxen #1 SMP Wed Apr 28
08:58:56 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

Everything works like a charm now.

Comment 17 Jarod Wilson 2010-05-28 15:21:49 UTC
in kernel-2.6.18-201.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

Comment 19 Siert Z. 2010-05-28 19:40:53 UTC
I was unable to boot the Xen kernel with the append dom0_mem=512M:                                                                  

  title Red Hat Enterprise Linux Server (2.6.18-201.el5xen)
    root (hd0,0)
    kernel /xen.gz-2.6.18-201.el5 dom0_mem=512M
    module /vmlinuz-2.6.18-201.el5xen ro root=/dev/vgsan/root

Rebooted and noticed that all other kernels failed with the same error:
    "Starting udev: Kernel panic - not syncing: Out of memory and no killable processes..."

Decided to change the append to dom0_mem=2048M and the problem doesn't occur any more.

Testing bandwidth to the Dom0 itself (10*100MB) from my desktop:

zijls@htn-ws-1376:~$ for i in {0..10}; do scp hsl0000:100MB /dev/null ; done
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    


Testing bandwith to DomU -guest- (10*100MB) from my desktop:

zijls@htn-ws-1376:~$ for i in {0..10}; do scp 10.10.4.7:100MB /dev/null ; done
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    
100MB                                               100%  100MB  11.1MB/s   00:09    

Also tested bandwidth from another physical server to the Xen guest. On the guest I started `nc -l 12345 >/dev/null` and on the physical machine a dd of 1GB over the wire:

[zijls@hsl2000 ~]$ dd if=/dev/zero bs=1024k count=1000 | nc 10.10.4.7 12345
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 14.7358 seconds, 71.2 MB/s

Everything still seems to be fine.

If any further testing is desired, don't hestitate to contact me.

PS: I created an SOS report and uploaded it to RHN, case 2022358.

Comment 20 Stanislaw Gruszka 2010-05-31 08:05:23 UTC
(In reply to comment #19)
> I was unable to boot the Xen kernel with the append dom0_mem=512M:              
> 
>   title Red Hat Enterprise Linux Server (2.6.18-201.el5xen)
>     root (hd0,0)
>     kernel /xen.gz-2.6.18-201.el5 dom0_mem=512M
>     module /vmlinuz-2.6.18-201.el5xen ro root=/dev/vgsan/root
> 
> Rebooted and noticed that all other kernels failed with the same error:
>     "Starting udev: Kernel panic - not syncing: Out of memory and no killable
> processes..."

IIRC this is regression, but seems to be unrelated with network drivers patches. Please open new bug report for xen component for that issue.

Comment 25 Douglas Silas 2010-06-28 20:26:57 UTC
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
Large-Receive Offload (LRO) is a performance optimization that enables the kernel to fetch and process, as a unit, more than one received packet from a network device. It was previously not possible to dynamically disable LRO for devices in a forwanding mode. This has been fixed with this update so that the kernel is able to dynamically disable LRO for devices in a forwarding state, or which had had LRO turned on manually.

Comment 28 Stanislaw Gruszka 2010-08-02 09:24:56 UTC
*** Bug 586352 has been marked as a duplicate of this bug. ***

Comment 29 Alan Pevec 2010-11-19 08:44:07 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Large-Receive Offload (LRO) is a performance optimization that enables the kernel to fetch and process, as a unit, more than one received packet from a network device. It was previously not possible to dynamically disable LRO for devices in a forwanding mode. This has been fixed with this update so that the kernel is able to dynamically disable LRO for devices in a forwarding state, or which had had LRO turned on manually.+Large-Receive Offload (LRO) is a performance optimization that enables the kernel to fetch and process, as a unit, more than one received packet from a network device. It was previously not possible to dynamically disable LRO for devices in a forwarding mode. This has been fixed with this update so that the kernel is able to dynamically disable LRO for devices in a forwarding state, or which had had LRO turned on manually.

Comment 33 Alan Pevec 2010-11-26 09:15:07 UTC
Comment on attachment 406910 [details]
bnx2x patch

Thanks for clarification Stanislaw!
Marking  patch in comment 4 obsolete to avoid further confusion.

Comment 34 Hangbin Liu 2010-12-03 07:43:34 UTC
verified bnx2x driver on kernel 2.6.18-235.el5

# uname -a
Linux hp-bl685cg6-01.rhts.eng.bos.redhat.com 2.6.18-235.el5 #1 SMP Wed Dec 1 12:27:10 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
# ethtool -i eth0
driver: bnx2x
version: 1.52.53-4
firmware-version: bc 4.8.0 phy baa0:0105
bus-info: 0000:03:00.0
# service network stop
# modprobe -r bnx2x
# modprobe bnx2x disable_tpa=0     
PCI: Enabling device 0000:03:00.0 (0040 -> 0042)
PCI: Enabling device 0000:03:00.1 (0040 -> 0042)
# service network start
# brctl addbr bridge
Bridge firewalling registered
# brctl addif bridge eth0
# dmesg | tail
...
Disabled lro on eth0
...

On kernel 2.6.18-194.el5 there is no such message


didn't test mlx4_en and s2io as there is no these patches .
enic have a  update driver version on kernel 2.6.18-227.el5 , replace LRO with GRO , needn't test too .

Comment 36 errata-xmlrpc 2011-01-13 21:27:23 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0017.html


Note You need to log in before you can comment on or make changes to this bug.