Description of problem: Attempt to create VLAN iface on bond of two adapters in two switch Then rmmod bonding,kernel panic. Version-Release number of selected component (if applicable): kernel 2.6.18-232.el5 How reproducible: Steps to Reproduce: 1.Configure bond0 in mode 6 of two ifaces 2. Create VLAN iface on bond0 3.rmmod bonding Actual results: [root@amd-ma78gm-02 ~]# modprobe -r bonding bonding: bond0: Warning: clearing HW address of bond0 while it still has VLANs. bonding: bond0: When re-adding slaves, make sure the bond's HW address matches its VLANs'. ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at drivers/net/bonding/bonding.h:135 invalid opcode: 0000 [1] SMP last sysfs file: /devices/pci0000:00/0000:00:00.0/irq CPU 2 Modules linked in: bonding radeon drm autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc cpufreq_ondemand powernow_k8 freq_table mperf 8021q ipv6 xfrm_nalgo crypto_api loop dm_multipath scsi_dh video backlight sbs power_meter i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac lp joydev shpchp snd_hda_intel snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc amd64_edac_mod snd_hwdep i2c_piix4 r8169 e1000 i2c_core snd mii floppy k10temp edac_mc tpm_tis parport_pc tpm soundcore ide_cd parport hwmon serio_raw cdrom tpm_bios pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ahci libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 2327, comm: modprobe Not tainted 2.6.18-232.el5 #1 RIP: 0010:[<ffffffff886fe742>] [<ffffffff886fe742>] :bonding:bond_vlan_rx_kill_vid+0xb1/0x1de RSP: 0000:ffff81020fb3ddf8 EFLAGS: 00010286 RAX: 00000000ffffffff RBX: ffff810210bb7000 RCX: 0000000000000000 RDX: 0000000000000055 RSI: 0000000000000c16 RDI: ffff810210bb7000 RBP: ffff810210bb7500 R08: ffff81022fe6b110 R09: 000000000000003c R10: ffffffff804af300 R11: 0000000000000000 R12: 0000000000000c16 R13: 0000000000000000 R14: 0000000000000c16 R15: 0000000000000c16 FS: 00002aef3f61a6e0(0000) GS:ffff810107b97e40(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000031840d4020 CR3: 000000020fe4a000 CR4: 00000000000006e0 Process modprobe (pid: 2327, threadinfo ffff81020fb3c000, task ffff810229248820) Stack: ffff810210bb7000 ffff810210bb7000 ffff81020fbc0000 0000000000000c16 0000000000000c16 ffff81020fbc0000 0000000000000000 ffffffff885b523a 0000000000000000 0000000000000006 ffff8102290c7800 ffff810210bb7000 Call Trace: [<ffffffff885b523a>] :8021q:unregister_vlan_dev+0x8f/0x119 [<ffffffff885b541a>] :8021q:vlan_device_event+0x156/0x17b [<ffffffff800675d4>] notifier_call_chain+0x20/0x32 [<ffffffff80233a88>] unregister_netdevice+0x17e/0x20c [<ffffffff886ffa2e>] :bonding:bond_free_all+0x3a/0xb5 [<ffffffff887098c8>] :bonding:bonding_exit+0x30/0x36 [<ffffffff800a8539>] sys_delete_module+0x196/0x1c5 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Code: 0f 0b 68 aa a4 70 88 c2 87 00 48 8d 7d 30 e8 1f 64 96 f7 48 RIP [<ffffffff886fe742>] :bonding:bond_vlan_rx_kill_vid+0xb1/0x1de RSP <ffff81020fb3ddf8> <0>Kernel panic - not syncing: Fatal exception Expected results: Additional info:
This appears to be a side-effect of the code added to enable netconsole over bonded interfaces as the BUG halt is from block_netpoll_tx. Assigning to Neil since that was his feature.
Created attachment 463769 [details] convert per cpu mask to a counter The more I look at it the more I think using a per-cpu flag was a mistake. The only time we query the flag is when we recurse through a netpoll path anyway, so the IFF_IN_NETPOLL flag gates us on same-cpu access anyway. And the possibility of sleeping in paths gives rise to the possibility of us clearing a different flag than we set, which is bad. There won't be any performance hit if we just make a counter out of this thing instead, and that will allow us to sleep while holding the netpoll_block_tx flag. It means that all netpoll clients will block at the same time, even if they could make forward progress, but no one actually has more than one netpoll client active at any one time, so thats not super relevant. Please test this patch out and let me know if it solves the problem. If it does I'll push it upstream and into RHEL5/6
Created attachment 463770 [details] updated patch to convert per-cpu flag to counter sorry, updated patch. Misnamed an origional file and so it didn't get picked up in the origional patch.
(In reply to comment #2) > Created attachment 463769 [details] > convert per cpu mask to a counter > > The more I look at it the more I think using a per-cpu flag was a mistake. The > only time we query the flag is when we recurse through a netpoll path anyway, > so the IFF_IN_NETPOLL flag gates us on same-cpu access anyway. And the > possibility of sleeping in paths gives rise to the possibility of us clearing a > different flag than we set, which is bad. There won't be any performance hit > if we just make a counter out of this thing instead, and that will allow us to > sleep while holding the netpoll_block_tx flag. It means that all netpoll > clients will block at the same time, even if they could make forward progress, > but no one actually has more than one netpoll client active at any one time, so > thats not super relevant. > > Please test this patch out and let me know if it solves the problem. If it > does I'll push it upstream and into RHEL5/6 This is probably a better solution than the CPU mask as it avoids the case where the locks are taken and released on a different CPU (though it seems those cases were removed). Should some care also be taken to make sure the ref counting is zero by the time the driver is removed?
We can certainly add a WARN_ON, but we should never be waiting on that refcount to decrement. the use paths of the counter are all within the driver, and the onus of making sure all the tasks/workqueues/etc which use this code are flushed is already part of the driver exit routine.
Something like a warning was all I was thinking about.
(In reply to comment #3) > Created attachment 463770 [details] > updated patch to convert per-cpu flag to counter > > sorry, updated patch. Misnamed an origional file and so it didn't get picked > up in the origional patch. This bug reproduces with probability. I service network restart about 20 times the kernel panic [root@ibm-ls21-03 network-scripts]# service network restart Shutting down interface bond0.10: Removed VLAN -:bond0.10:- [ OK ] Shutting down interface bond0: bonding: bond0: Warning: the permanent HWaddr of eth0 - 00:14:5E:6D:1C:B8 - is still in use by bond0. Set the HWaddr of eth0 to a different address to avoid conflicts. ----------- [cut here ] --------- [please bite here ] --------- Kernel BUG at drivers/net/bonding/bonding.h:135 invalid opcode: 0000 [1] SMP last sysfs file: /class/net/bond0/bonding/slaves CPU 0 Modules linked in: bonding 8021q autofs4 hidp rfcomm l2cap bluetooth lockd sunrpc ipv6 xfrm_nalgo crypto_api loop dm_multipath scsi_dh video backlight sbs power_meter i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sg i2c_piix4 tpm_tis k8temp i2c_core k8_edac bnx2 tpm hwmon edac_mc serio_raw tpm_bios pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod shpchp mptsas mptscsih mptbase scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 12674, comm: ifdown-eth Not tainted 2.6.18-232.el5 #1 RIP: 0010:[<ffffffff884c2c0b>] [<ffffffff884c2c0b>] :bonding:bond_release+0x62/0x4f1 RSP: 0018:ffff810127609e28 EFLAGS: 00010286 RAX: 00000000ffffffff RBX: 00000000000005dc RCX: ffffffff80318f28 RDX: ffffffff80318f28 RSI: ffff81022c488000 RDI: ffff8101281f2530 RBP: ffff8101281f2500 R08: ffffffff80318f28 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000080 R12: ffff8101281f2000 R13: 0000000000000006 R14: ffff81022c488000 R15: ffff81012ea50ac0 FS: 00002addc8667f50(0000) GS:ffffffff80424000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 000000388ca69220 CR3: 000000012743a000 CR4: 00000000000006e0 Process ifdown-eth (pid: 12674, threadinfo ffff810127608000, task ffff810128bc57a0) Stack: 00000000000080d0 ffffffff8006456b ffff810128bc57a0 00000000000005dc ffff81022c488000 ffff8101281f2500 0000000000000006 0000000000000006 ffff81012ea50ac0 ffffffff884cbb54 000000316874652d 0000000000000000 Call Trace: [<ffffffff8006456b>] __down_write_nested+0x12/0x92 [<ffffffff884cbb54>] :bonding:bonding_store_slaves+0x25c/0x2f7 [<ffffffff8010fdb5>] sysfs_write_file+0xb9/0xe8 [<ffffffff80016af0>] vfs_write+0xce/0x174 [<ffffffff800173a8>] sys_write+0x45/0x6e [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Code: 0f 0b 68 aa d4 4c 88 c2 87 00 4c 8b 6d 08 31 c0 eb 0c 4d 39 RIP [<ffffffff884c2c0b>] :bonding:bond_release+0x62/0x4f1 RSP <ffff810127609e28> <0>Kernel panic - not syncing: Fatal exception It looks like the issue is not the same.
I test on kernel 2.6.18-194.el5 It does not reproduce. I think it is a regression bug
Ok, I'm going to post this upstream and backport it then
*** This bug has been marked as a duplicate of bug 659594 ***