Bug 818504

Summary: Disable LRO for all NICs that have LRO enabled [rhel-6.2.z]
Product: Red Hat Enterprise Linux 6 Reporter: RHEL Program Management <pm-rhel>
Component: kernelAssignee: Frantisek Hrbata <fhrbata>
Status: CLOSED ERRATA QA Contact: Liang Zheng <lzheng>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.3CC: acathrow, agospoda, apevec, atzhang, bsarathy, cpelland, dhoward, djuran, dledford, dyasny, fyu, gouyang, jboggs, jturner, kzhang, leiwang, llim, lzheng, mburns, moli, mwagner, nhorman, ovirt-maint, pcao, plundin, plyons, pm-eus, sforsber, sghosh, tvvcox, vbian, ycui, yeylon, zhchen
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.32-220.20.1.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-18 13:34:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 772317    
Bug Blocks:    

Description RHEL Program Management 2012-05-03 08:50:57 UTC
This bug has been copied from bug #772317 and has been proposed
to be backported to 6.2 z-stream (EUS).

Comment 6 Zhenjie Chen 2012-06-12 10:43:35 UTC
I think this patch doesn't fix the bug and introduces a regression bug.
See https://bugzilla.redhat.com/show_bug.cgi?id=794647

reproduce:

[root@hp-dl585g7-02 ~]#  uname -r
2.6.32-220.22.1.el6.x86_64
[root@hp-dl585g7-02 ~]# ethtool -i eth0
driver: netxen_nic
[root@hp-dl585g7-02 ~]# ethtool -k eth0 |grep large
large-receive-offload: on
[root@hp-dl585g7-02 ~]# modprobe bonding mode=1 miimon=100
[root@hp-dl585g7-02 ~]# ifconfig bond0 up
[root@hp-dl585g7-02 ~]#  ifenslave bond0 eth0

modprobe bonding mode=1 ethtool -k eth0 |grep large
large-receive-offload: on
[root@hp-dl585g7-02 ~]# ethtool -k bond0 |grep large
large-receive-offload: on

[root@hp-dl585g7-02 ~]#  brctl addbr br0
[root@hp-dl585g7-02 ~]# brctl addif br0 bond0
device bond0 entered promiscuous mode
device eth0 entered promiscuous mode
------------[ cut here ]------------
WARNING: at net/core/dev.c:1234 dev_disable_lro+0x7b/0x80() (Tainted: G           ---------------- T)
Hardware name: ProLiant DL585 G7
Modules linked in: bridge stp llc bonding sunrpc ipv6 power_meter bna(T) cxgb4 hpilo hpwdt sfc mdio netxen_nic microcode serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 sg shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom hpsa pata_acpi ata_generic pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_tgt]
Pid: 12312, comm: brctl Tainted: G           ---------------- T 2.6.32-220.22.1.el6.x86_64 #1
Call Trace:
 [<ffffffff81069c97>] ? warn_slowpath_common+0x87/0xc0
 [<ffffffff81069cea>] ? warn_slowpath_null+0x1a/0x20
 [<ffffffff8142a8db>] ? dev_disable_lro+0x7b/0x80
 [<ffffffffa050030e>] ? br_add_if+0x23e/0x3e0 [bridge]
 [<ffffffffa0500a58>] ? add_del_if+0x78/0xa0 [bridge]
 [<ffffffffa050138b>] ? br_dev_ioctl+0x4b/0x90 [bridge]
 [<ffffffff8142fcde>] ? dev_ifsioc+0xfe/0x2b0
 [<ffffffff814301d3>] ? dev_ioctl+0x343/0x5e0
 [<ffffffff812158a4>] ? inode_has_perm+0x54/0xa0
 [<ffffffff81418d0d>] ? sock_ioctl+0x9d/0x280
 [<ffffffff811897a2>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff81189944>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff81189ec1>] ? sys_ioctl+0x81/0xa0
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
---[ end trace 995ffc3c7c71d89a ]---

[root@hp-dl585g7-02 ~]# ethtool -k bond0 |grep large
large-receive-offload: on

Comment 7 Neil Horman 2012-06-12 11:00:11 UTC
you also need to pull back commit 3ef3cd0b54f305977f07b3ee8067f72ad96d2acd
from the 6.3 tree to avoid this

Comment 9 Zhenjie Chen 2012-06-15 04:11:37 UTC
verified on kernel 2.6.32-220.23.1.el6.x86_64

Step A:
[root@hp-nehalem-02 ~]#  echo 1 > /proc/sys/net/ipv4/ip_forward
[root@hp-nehalem-02 ~]#  modprobe bonding mode=1 miimon=100
normal, no Call Trace

Step B:
[root@hp-nehalem-02 ~]#  modprobe bonding mode=1 miimon=100
[root@hp-nehalem-02 ~]# ifconfig bond0 up
[root@hp-nehalem-02 ~]# ifenslave bond0 eth0
[root@hp-nehalem-02 ~]# brctl addbr br0
[root@hp-nehalem-02 ~]# brctl addif br0 bond0
normal, no Call Trace

[root@hp-nehalem-02 ~]# ethtool -k bond0 |grep large
large-receive-offload: off
right, bond0's lro auto be set  off

Step C: performance
from another machine which in the same network
dd if=/dev/zero of=bigfile bs=2G count=1
[root@hp-dl585-02 ~]# scp bigfile hp-nehalem-02.rhts.eng.bos.redhat.com:
bigfile                                       100% 2048MB  33.6MB/s   01:01   

[root@hp-dl585-02 ~]# netperf -H  hp-nehalem-02.rhts.eng.bos.redhat.com -l 7200
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.16.65.145 (10.16.65.145) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    7200.03    931.44  

-----------------
everything is OK
set verified

Comment 11 errata-xmlrpc 2012-06-18 13:34:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0743.html