Bug 578531
| Summary: | [RHEL5.5] soft lockup on vlan with bonding in balance-alb mode | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Yury Konovalov <YKonovalov> |
| Component: | kernel | Assignee: | Andy Gospodarek <agospoda> |
| Status: | CLOSED ERRATA | QA Contact: | Liang Zheng <lzheng> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 5.5 | CC: | agospoda, benj, benlu, bshepher, dhoward, dmoore, fleitner, gianluca.cecchi, govind.rhul, haliu, herbert.xu, hjia, jaeshin, jolsa, jpirko, justdave, jwest, khorenko, kzhang, liko, lzheng, mzeier, nenad, nhorman, pep, peterm, plsmith, roy.keene, schlegel, sgruszka, shyam, Stuart.Kirk, tao, tgraf, villapla |
| Target Milestone: | rc | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
An attempt to create a VLAN interface on a bond of two bnx2 adapters in two switch configurations resulted in a soft lockup after a few seconds. This was caused by an incorrect use of a bonding pointer. With this update, soft lockups no longer occurs and creating a VLAN interface works as expected.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-01-13 21:23:29 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 630540, 640803 | ||
| Attachments: | |||
|
Description
Yury Konovalov
2010-03-31 14:53:50 UTC
We are experiencing the same problem on multiple hardware platforms: Kernel 2.6.18-194.el5 - bnx2 (2.0.8b) - e1000 (7.3.21-k4-NAPI) BUG: soft lockup - CPU#7 stuck for 10s! [swapper:0] CPU 7: Modules linked in: 8021q bridge autofs4 lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport st e1000e serio_raw hpilo shpchp bnx2(U) pcspkr sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 0, comm: swapper Tainted: G 2.6.18-194.el5 #1 RIP: 0010:[<ffffffff80065c20>] [<ffffffff80065c20>] .text.lock.spinlock+0x26/0x30 RSP: 0018:ffff81031feb3cb0 EFLAGS: 00000286 RAX: ffff81031feabfd8 RBX: ffff8103125b16c0 RCX: ffff8103125b1000 RDX: ffff81031b016710 RSI: ffff8103125b1000 RDI: ffff8103125b1758 RBP: ffff81031feb3c30 R08: ffff8103176f2e80 R09: ffff810316545180 R10: ffff81031feb3db8 R11: 00000000000000c8 R12: ffffffff8005ec8e R13: ffff8103125b1758 R14: ffffffff8007922b R15: ffff81031feb3c30 FS: 0000000000000000(0000) GS:ffff81031fe283c0(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 000000001efbb670 CR3: 0000000310e8a000 CR4: 00000000000006e0 Call Trace: <IRQ> [<ffffffff884ce002>] :bonding:rlb_arp_recv+0xb2/0x146 [<ffffffff800209ba>] netif_receive_skb+0x43e/0x49f [<ffffffff882abe50>] :bnx2:bnx2_poll+0x1245/0x14e3 [<ffffffff80151248>] __next_cpu+0x19/0x28 [<ffffffff8008ccb0>] find_busiest_group+0x20d/0x621 [<ffffffff800c907c>] free_pages_bulk+0x1f0/0x268 [<ffffffff8000c88a>] net_rx_action+0xac/0x1e0 [<ffffffff80012409>] __do_softirq+0x89/0x133 [<ffffffff8005f2fc>] call_softirq+0x1c/0x28 [<ffffffff8006dba8>] do_softirq+0x2c/0x85 [<ffffffff8006da30>] do_IRQ+0xec/0xf5 [<ffffffff8005e615>] ret_from_intr+0x0/0xa <EOI> [<ffffffff8019e040>] acpi_processor_idle_simple+0x17d/0x30e [<ffffffff8019df2f>] acpi_processor_idle_simple+0x6c/0x30e [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e [<ffffffff800497be>] cpu_idle+0x95/0xb8 [<ffffffff80078997>] start_secondary+0x498/0x4a7 BUG: soft lockup - CPU#3 stuck for 10s! [swapper:0] CPU 3: Modules linked in: 8021q bridge autofs4 lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport st sg hpilo shpchp e1000e pcspkr bnx2(U) serio_raw dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 0, comm: swapper Tainted: G 2.6.18-194.el5 #1 RIP: 0010:[<ffffffff80065c20>] [<ffffffff80065c20>] .text.lock.spinlock+0x26/0x30 RSP: 0018:ffff81010afbfd60 EFLAGS: 00000286 RAX: ffff81010afb9fd8 RBX: ffff8103170826c0 RCX: ffff810317082000 RDX: ffff8103152f1710 RSI: ffff810317082000 RDI: ffff810317082758 RBP: ffff81010afbfce0 R08: 0000000000000000 R09: 0000000000000000 R10: ffff8103153d95c0 R11: 00000000000000c8 R12: ffffffff8005ec8e R13: ffff810317082758 R14: ffffffff8007922b R15: ffff81010afbfce0 FS: 0000000000000000(0000) GS:ffff81031ff236c0(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000003516003080 CR3: 0000000000201000 CR4: 00000000000006e0 Call Trace: <IRQ> [<ffffffff884c7002>] :bonding:rlb_arp_recv+0xb2/0x146 [<ffffffff800209ba>] netif_receive_skb+0x43e/0x49f [<ffffffff882e33cd>] :e1000e:e1000_receive_skb+0x1b5/0x1d6 [<ffffffff882e7b27>] :e1000e:e1000_clean_rx_irq+0x27a/0x321 [<ffffffff882e5bc5>] :e1000e:e1000_clean+0x7c/0x29a [<ffffffff8000c88a>] net_rx_action+0xac/0x1e0 [<ffffffff882e5a55>] :e1000e:e1000_intr_msi+0xd6/0xe0 [<ffffffff80012409>] __do_softirq+0x89/0x133 [<ffffffff8005f2fc>] call_softirq+0x1c/0x28 [<ffffffff8006dba8>] do_softirq+0x2c/0x85 [<ffffffff8006da30>] do_IRQ+0xec/0xf5 [<ffffffff8005e615>] ret_from_intr+0x0/0xa <EOI> [<ffffffff8019e040>] acpi_processor_idle_simple+0x17d/0x30e [<ffffffff8019df2f>] acpi_processor_idle_simple+0x6c/0x30e [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e [<ffffffff8019dec3>] acpi_processor_idle_simple+0x0/0x30e [<ffffffff800497be>] cpu_idle+0x95/0xb8 [<ffffffff80078997>] start_secondary+0x498/0x4a7 I'm trying to reproduce this on a machine running 2.6.18-194.3.1.el5 and can't. Yury, do you still have that problem if you upgrade to 2.6.18-194.3.1.el5? I confirm to have this problem with 2.6.18-194.3.1.el5 on a Dell PowerEdge 1650 with two e1000 NICs. lspci output: 00:00.0 Host bridge: Broadcom CNB20HE Host Bridge (rev 23) 00:00.1 Host bridge: Broadcom CNB20HE Host Bridge (rev 01) 00:00.2 Host bridge: Broadcom CNB20HE Host Bridge (rev 01) 00:00.3 Host bridge: Broadcom CNB20HE Host Bridge (rev 01) 00:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93) 00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93) 00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05) 00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge 01:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02) 01:04.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02) 01:08.0 PCI bridge: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge (rev 01) 01:08.1 RAID bus controller: Dell PowerEdge Expandable RAID Controller 3/Di (rev 01) Hotfix: change to bonding mode 5. I'm experiencing the same problem on a Dell 2950 and 2.6.18-194.3.1.el5 with two Intel and two Broadcom (the embedded ones) adapters. I have bonding and VLANs too. Updated (via scratch install) from rh el 4.5 x86_64 to rh el 5.5 (+updates till today) and I'm experiencing these problems that before I had not. Changing mode from 6 to 5 workarounds the problem for me too. # lspci|grep thern 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) 09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) 0a:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) 0a:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06) I get the same stack trace (I'm going to attach screenshot) both with bonding done on top of a pairs of bnx2 and on top of a pair of e1000e ones. So it is driver independent Created attachment 423266 [details]
stack trace with bonding on a paier of intel cards
I had this issue when I switched to mode 6 (alb-balance). BUG: soft lockup - CPU#1 stuck for 10s! [swapper:0] Pid: 0, comm: swapper EIP: 0060:[<c061dc4c>] CPU: 1 EIP is at _spin_lock_bh+0xd/0x18 EFLAGS: 00000286 Not tainted (2.6.18-194.3.1.el5 #1) EAX: c0749000 EBX: f69a3d50 ECX: f68a8d2c EDX: f69a3800 ESI: f69a3cfc EDI: f69a3d50 EBP: f68fd600 DS: 007b ES: 007b CR0: 8005003b CR2: b7fc8000 CR3: 00742000 CR4: 000006d0 [<f8fed0ea>] rlb_arp_recv+0x98/0x11d [bonding] [<c05c0aa8>] netif_receive_skb+0x3ac/0x401 [<f8ba5df3>] tg3_poll+0x64f/0xc28 [tg3] [<c041eeda>] __activate_task+0x4a/0x59 [<c04ee889>] rb_erase+0x176/0x22f [<c05c2995>] net_rx_action+0x9c/0x1a7 [<c042a377>] __do_softirq+0x87/0x114 [<c04073cf>] do_softirq+0x52/0x9c [<c044f158>] __do_IRQ+0x0/0xd6 [<c04074ce>] do_IRQ+0xb5/0xc3 [<c0405946>] common_interrupt+0x1a/0x20 [<c0403bb0>] default_idle+0x0/0x59 [<c0403be1>] default_idle+0x31/0x59 [<c0403ca8>] cpu_idle+0x9f/0xb9 ======================= I can also confirm similar behavior on Cisco UCS B250-M2 blade system using the ixgbe network driver. In our situation we had physical eth0, eth1, eth2, eth3 which we were using mode 6 to create a bond0. As soon as the bond0.x interface was brought up during boot the boot process is halted. If done via command line, a stack trace similar to the above is produced. Unfortunately the soft-lockups don't really tell me much as it only shows the receive patch not being able to take the lock. I'll have to try and reproduce this myself. The CPU is stuck at rlb_update_entry_from_arp() trying to get the lock _lock_rx_hashtbl(bond) and for some reason, it's unable to get it, so the watchdog fires showing that back trace. However, this is a consequence and not a root cause because, in this case, another CPU is holding that lock leaving others waiting for it long enough to trigger the watchdog. Therefore, can you get few outputs of sysrq+t and sysrq+w while the problem is happening? It could point to us which CPU is holding the lock and why. thanks, fbl I could reproduce this here, so no need to provide sysrq+t or sysrq+w. Flavio, can you post any new information you have? Created attachment 432903 [details]
suggested patch
Hi Andy,
The problem has been introduced by the following patch:
[net] bonding: allow arp_ip_targets on separate vlan from bond device
and not fixed by the later patch:
[net] fixup problems with vlans and bonding
The problem happens because in rlb_arp_recv(), the struct bonding *bond
pointer is a vlan's net_device struct instead, so it can either oops or
just hangs on a invalid spinlock.
I can reproduce both situations following the instructions in the
ticket's summary.
The upstream fixes rlb_arp_recv() to look for the flag IFF_802_1Q_VLAN
and if it is present, then find the underlying bonding device.
I have the patch backported and it works out on my tests.
Please review.
fbl
I don't think this can count as a regression. The upstream patch below that adds the code described above was added in 2008 and were it not for the code that added support for arp_ip_targets on a separate VLAN this problem would have never been seen. Here is the upstream patch we should consider:
commit 6146b1a4da98377e4abddc91ba5856bef8f23f1e
Author: Jay Vosburgh <fubar.com>
Date: Tue Nov 4 17:51:15 2008 -0800
bonding: Fix ALB mode to balance traffic on VLANs
This does not diminish the importance, but finding out that something is broken after a feature was added is not a regression.
*** Bug 615996 has been marked as a duplicate of this bug. *** Created attachment 432929 [details]
bonding-fix-alb-mode-to-balance-traffic-on-vlans.patch
This patch should resolve the issue based on Flavio's analysis. It is a backport of upstream commits:
6146b1a4da98377e4abddc91ba5856bef8f23f1e bonding: Fix ALB mode to balance traffic on VLANs
2690f8d62e98779c71625dba9a0fd525d8b2263d bonding: Remove debug printk
I have added this to my rhel5 gtest repo and will post to this bug when new test kernels are available.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. I thought it was a regression because before vlan over bonding was working and after adding a new feature, the same setup doesn't work anymore. Regarding to the new patch, the next merges will be a bit more complicated because the loop is not using vlan_dev_real_dev() as upstream does, so I have added a macro to deal with it. Also, I intentionally removed pk_type->dev initialization because it didn't make sense to my eyes for this kernel version, but I could be overlooking something. (In reply to comment #18) > Also, I intentionally removed pk_type->dev > initialization because it didn't make sense to my eyes for this kernel version, > but I could be overlooking something. Removing it was a good idea. It was removed today as it caused a panic in a multiple bond configuration. (In reply to comment #19) > Removing it was a good idea. It was removed today as it caused a panic in a > multiple bond configuration. Yeah, let me know when you have the final patch ready. Created attachment 434836 [details]
bonding-fix-alb-mode-to-balance-traffic-on-vlans-updated.patch
Here is an updated patch. Feedback is welcome.
My test kernels have been updated to include a patch for this bugzilla. http://people.redhat.com/agospoda/#rhel5 Please test them and report back your results. (In reply to comment #27) > My test kernels have been updated to include a patch for this bugzilla. > > http://people.redhat.com/agospoda/#rhel5 > > Please test them and report back your results. We've been running this for a couple of weeks on one of our production boxes and haven't seen any issues whatsoever. # uptime 20:39:44 up 13 days, 20:30, 1 user, load average: 1.30, 1.52, 1.56 # uname -a Linux foo.mozilla.net 2.6.18-212.el5.gtest.89 #1 SMP Mon Aug 16 14:01:15 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux Glad to hear that, Shyam! I appreciate the feedback. Andy, Finnaly, I got a change to put 2.6.18-212.el5.gtest.89 under a test on a IBM HS21 system with two nics in two switch configuration. Works smoothly for about half a day and perform pretty good (~ 1883 Mbits/sec on multi-node iperf test). Thanks for the patch and test kernel package. It seems like this bug is fixed. Andy, Could you please share kernel-headers package of your test kernel build? I would like to test GPFS filesystem with bonding on your test kernel and I need to build IBM gpfs modules and some other kernel-dependent staff. kernel-devel is not enough for me. Glad to know they are working, Yury. Thanks for that feedback. I also added all of the headers rpms to my people page here: http://people.redhat.com/agospoda/#rhel5 Please let me know if any other rpms from that build would be helpful and are not on my people page. Thank you, Andy for making headers package available. I will report if any of my ongoing stress tests fail. Until now it works and perform perfectly. in kernel-2.6.18-219.el5 You can download this test kernel from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. It's ok also in my Dell 2950 with kernel 2.6.18-219.el5 and alb bonding and some vlans over bonding of two Broadcom adapters: 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) 09:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12) At least in half an hour no error at all. In dmesg: Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008) bonding: In ALB mode you might experience client disconnections upon reconnection of a link if the bonding module updelay parameter (200 msec) is incompatible with the forwarding delay time of the switch bonding: MII link monitoring set to 100 ms ADDRCONF(NETDEV_UP): bond0: link is not ready bonding: bond0: Adding slave eth0. bnx2: eth0: using MSI bonding: bond0: enslaving eth0 as an active interface with a down link. bonding: bond0: Adding slave eth1. bnx2: eth1: using MSI bonding: bond0: enslaving eth1 as an active interface with a down link. bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex bonding: bond0: link status up for interface eth0, enabling it in 0 ms. bonding: bond0: link status definitely up for interface eth0. bonding: bond0: making interface eth0 the new active one. bonding: bond0: first active interface up! ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex bonding: bond0: link status up for interface eth1, enabling it in 200 ms. bonding: bond0: link status definitely up for interface eth1. 802.1Q VLAN Support v1.8 Ben Greear ... All bugs added by David S. Miller ... bond0: no IPv6 routers present bond0.13: no IPv6 routers present bond0.139: no IPv6 routers present bond0.221: no IPv6 routers present bond0.66: no IPv6 routers present bond0.68: no IPv6 routers present bond0.800: no IPv6 routers present Gianluca Excellent! Thanks for the feedback, Gianluca. (In reply to comment #40) > in kernel-2.6.18-219.el5 > You can download this test kernel from http://people.redhat.com/jwilson/el5 > > Detailed testing feedback is always welcomed. # uname -a Linux foo.mozilla.net 2.6.18-219.el5 #1 SMP Thu Sep 9 17:10:23 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux # uptime 07:41:25 up 9 days, 10:42, 1 user, load average: 4.77, 4.59, 4.59 All good, same production server as in comment #31 and seems to be fine. *** Bug 602197 has been marked as a duplicate of this bug. *** This patch works for us. Can you please let us know when this patch will be available in redhat-release, as it fails on latest kernel also 2.6.18-194.11.4.el5 Thanks (In reply to comment #46) > This patch works for us. Can you please let us know when this patch will be > available in redhat-release, as it fails on latest kernel also > 2.6.18-194.11.4.el5 > > Thanks Right now it will not be available until the official RHEL5.6 update ships. As of today, I cannot say for sure when RHEL5.6 will ship. If you need support in a RHEL5.5 update kernel, please go through our support portal: https://access.redhat.com/home and request that this patch is added. You can reference this bugzilla if needed. I DO have a case opened for this bug and latest answer has been (right today after asking for a date):
> Any information about an official and supported update containing the fix?
At this moment the errata which will contain the fix is not yet confirmed.
My case is
Case Number : 00332049
Case Open Date : 2010-06-11 14:42:07
If you see the date, we are about 3 months and no official solution yet....
So the question is: what is the added value of active subscription?
Excuse me for being a little sarcastic...
Gianluca
(In reply to comment #48) > I DO have a case opened for this bug and latest answer has been (right today > after asking for a date): > > Any information about an official and supported update containing the fix? > At this moment the errata which will contain the fix is not yet confirmed. > > My case is > Case Number : 00332049 > Case Open Date : 2010-06-11 14:42:07 > > If you see the date, we are about 3 months and no official solution yet.... > So the question is: what is the added value of active subscription? > > Excuse me for being a little sarcastic... > > Gianluca I'm glad you opened a support ticket for this. Opening tickets is really the only way for us to know that this fix is critical enough to paying customers that we need to add it to the currently shipping (in this case 2.6.18-194) kernel stream. Thanks for doing that. It can be a bit confusing, but we open a bug for each release that requires a patch. This bug will address the problem on upcoming RHEL5.6 and bug 630540 will address the problem on already released RHEL5.5 since there was enough customer demand (not just noise in bugzilla) to fix it before RHEL5.6 shipped. I made some noise over in bug 630540, so hopefully things will move forward with a fix in RHEL5.5. (In reply to comment #49) > I made some noise over in bug 630540, so hopefully things will move forward > with a fix in RHEL5.5. I saw it, thanks. I (we) keep on waiting for a fix, then
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
An attempt to create a VLAN interface on a bond of two bnx2 adapters in two switch configurations resulted in a soft lockup after a few seconds. This was caused by an incorrect use of a bonding pointer. With this update, soft lockups no longer occurs and creating a VLAN interface works as expected.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0017.html |