Hide Forgot
Description of problem: Run testing on two bonding with arp monitor,when remove the bonding module,the kernel hungs and cannot reboot. Version-Release number of selected component (if applicable): kernel 2.6.32-166.el6 How reproducible: always Steps to Reproduce: My reproducerifconfig eth4 up ifconfig eth5 up ifconfig eth6 up ifconfig eth7 up sleep 10 modprobe bonding mode=1 arp_interval=1000 arp_ip_target=192.168.1.21 max_bonds=2 ifconfig bond0 192.168.1.10/24 up ifconfig bond1 192.168.1.11/24 up ifenslave bond0 eth4 eth5 ifenslave bond1 eth6 eth7 sleep 10 modprobe -r bonding Actual results: [root@hp-dl580g7-01 ~]# Broadcast message from root.eng.nay.redhat.com (/dev/pts/0) at 5:46 ... The system is going down for reboot NOW! INFO: task events/20:119 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. events/20 D 0000000000000014 0 119 2 0x00000000 ffff8802366e7d80 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 ffff8802366de6b8 ffff8802366e7fd8 000000000000f598 ffff8802366de6b8 Call Trace: [<ffffffff814dc84e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff8142ea60>] ? linkwatch_event+0x0/0x30 [<ffffffff814dc6eb>] mutex_lock+0x2b/0x50 [<ffffffff8142b985>] rtnl_lock+0x15/0x20 [<ffffffff8142ea6e>] linkwatch_event+0xe/0x30 [<ffffffff81088830>] worker_thread+0x170/0x2a0 [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810886c0>] ? worker_thread+0x0/0x2a0 [<ffffffff8108ddf6>] kthread+0x96/0xa0 [<ffffffff8100c1ca>] child_rip+0xa/0x20 [<ffffffff8108dd60>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 INFO: task ntpd:2650 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. ntpd D 0000000000000006 0 2650 1 0x00000080 ffff880235331cc8 0000000000000086 ffff880235331c68 ffffffff8105f0c4 0000000000000000 ffff8802366b75b8 ffff880247415f80 ffff8802366b7580 ffff880234aa5078 ffff880235331fd8 000000000000f598 ffff880234aa5078 Call Trace: [<ffffffff8105f0c4>] ? check_preempt_wakeup+0x1b4/0x290 [<ffffffff814dc84e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff814dc6eb>] mutex_lock+0x2b/0x50 [<ffffffff8142b985>] rtnl_lock+0x15/0x20 [<ffffffff81421665>] dev_ioctl+0x125/0x5d0 [<ffffffff8120e174>] ? inode_has_perm+0x54/0xa0 [<ffffffff8115aaf5>] ? kmem_cache_alloc_notrace+0x115/0x130 [<ffffffff8140b1dd>] sock_ioctl+0x9d/0x280 [<ffffffff81184ee2>] vfs_ioctl+0x22/0xa0 [<ffffffff81185084>] do_vfs_ioctl+0x84/0x580 [<ffffffff81185601>] sys_ioctl+0x81/0xa0 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b INFO: task mingetty:2786 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 0000000000000004 0 2786 1 0x00000084 ffff88023555f898 0000000000000046 0000000000000000 0000004000000002 ffff88023555f8c8 ffffffff81165e58 0000000200012e60 000000010003bf25 ffff880235327078 ffff88023555ffd8 000000000000f598 ffff880235327078 Call Trace: [<ffffffff81165e58>] ? __mem_cgroup_try_charge+0x78/0x420 [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff811674c5>] ? mem_cgroup_charge_common+0xa5/0xd0 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107a17b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff8130960e>] ? tty_ldisc_deref+0xe/0x10 [<ffffffff81300b20>] ? tty_read+0xb0/0xf0 [<ffffffff81172f85>] ? vfs_read+0xb5/0x1a0 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task mingetty:2788 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 0000000000000002 0 2788 1 0x00000084 ffff88043569f898 0000000000000046 0000000000000000 ffffffff00000002 ffff88043569f8c8 ffffffff81165e58 ffff88043569f838 000000010003bf25 ffff880433687af8 ffff88043569ffd8 000000000000f598 ffff880433687af8 Call Trace: [<ffffffff81165e58>] ? __mem_cgroup_try_charge+0x78/0x420 [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50 [<ffffffff810537e4>] ? find_busiest_group+0x244/0xb20 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107a17b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff8130960e>] ? tty_ldisc_deref+0xe/0x10 [<ffffffff81300b20>] ? tty_read+0xb0/0xf0 [<ffffffff81172f85>] ? vfs_read+0xb5/0x1a0 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task mingetty:2790 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 000000000000000e 0 2790 1 0x00000084 ffff880235b69898 0000000000000046 0000000000000000 0000000000000002 ffff880235b698c8 ffffffff81165e58 ffff880200000000 000000010003bf27 ffff880235326638 ffff880235b69fd8 000000000000f598 ffff880235326638 Call Trace: [<ffffffff81165e58>] ? __mem_cgroup_try_charge+0x78/0x420 [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50 [<ffffffff81053f4a>] ? find_busiest_group+0x9aa/0xb20 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107a17b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff8130960e>] ? tty_ldisc_deref+0xe/0x10 [<ffffffff81300b20>] ? tty_read+0xb0/0xf0 [<ffffffff81172f85>] ? vfs_read+0xb5/0x1a0 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task mingetty:2792 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 0000000000000003 0 2792 1 0x00000084 ffff880233181898 0000000000000046 0000000000000000 0000000000000002 ffff8802331818c8 ffffffff81165e58 ffff880200000000 000000010003bf25 ffff8802331286b8 ffff880233181fd8 000000000000f598 ffff8802331286b8 Call Trace: [<ffffffff81165e58>] ? __mem_cgroup_try_charge+0x78/0x420 [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff811674c5>] ? mem_cgroup_charge_common+0xa5/0xd0 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107a17b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff8130960e>] ? tty_ldisc_deref+0xe/0x10 [<ffffffff81300b20>] ? tty_read+0xb0/0xf0 [<ffffffff81172f85>] ? vfs_read+0xb5/0x1a0 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task mingetty:2797 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 0000000000000014 0 2797 1 0x00000084 ffff880234749898 0000000000000046 0000000000000000 0000000000000002 ffff8802347498c8 ffffffff81165e58 ffff880200000000 000000010003bf39 ffff880234801a78 ffff880234749fd8 000000000000f598 ffff880234801a78 Call Trace: [<ffffffff81165e58>] ? __mem_cgroup_try_charge+0x78/0x420 [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff812628a9>] ? cpumask_next_and+0x29/0x50 [<ffffffff810537e4>] ? find_busiest_group+0x244/0xb20 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107a17b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff8130960e>] ? tty_ldisc_deref+0xe/0x10 [<ffffffff81300b20>] ? tty_read+0xb0/0xf0 [<ffffffff81172f85>] ? vfs_read+0xb5/0x1a0 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task bash:3120 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. bash D 0000000000000002 0 3120 1 0x00000080 ffff8802264dd898 0000000000000046 0000000000000000 0000000000012b40 0000000040021b40 0000000000000000 ffff880236fdfd50 000000010003bf3f ffff8802265d3af8 ffff8802264ddfd8 000000000000f598 ffff8802265d3af8 Call Trace: [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff811da6b4>] ? free_proc_entry+0x74/0x90 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff810894b4>] flush_workqueue+0x54/0x80 [<ffffffff810894f5>] flush_scheduled_work+0x15/0x20 [<ffffffff813095ac>] tty_ldisc_release+0x3c/0x90 [<ffffffff8130383b>] tty_release_dev+0x40b/0x5e0 [<ffffffff8112eece>] ? __dec_zone_page_state+0x2e/0x30 [<ffffffff81303a2e>] tty_release+0x1e/0x30 [<ffffffff81173ff5>] __fput+0xf5/0x210 [<ffffffff81174135>] fput+0x25/0x30 [<ffffffff8116f6ad>] filp_close+0x5d/0x90 [<ffffffff81069c8f>] put_files_struct+0x7f/0xf0 [<ffffffff81069d53>] exit_files+0x53/0x70 [<ffffffff8106bdc5>] do_exit+0x185/0x860 [<ffffffff8107d94d>] ? __sigqueue_free+0x3d/0x50 [<ffffffff8108126f>] ? __dequeue_signal+0xdf/0x1f0 [<ffffffff8106c4f8>] do_group_exit+0x58/0xd0 [<ffffffff81081946>] get_signal_to_deliver+0x1f6/0x460 [<ffffffff8100a365>] do_signal+0x75/0x800 [<ffffffff8107ffef>] ? kill_pid_info+0x3f/0x60 [<ffffffff810d180b>] ? audit_syscall_exit+0x25b/0x290 [<ffffffff8100ab80>] do_notify_resume+0x90/0xc0 [<ffffffff8100b440>] int_signal+0x12/0x17 INFO: task bond1:18321 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. bond1 D 000000000000000e 0 18321 2 0x00000080 ffff8802353c5d20 0000000000000046 0000000000000000 ffffffff81103c56 ffff8802265d2b00 ffff880236ffaa80 ffff8802353c5d70 00000001000397b6 ffff8802265d30b8 ffff8802353c5fd8 000000000000f598 ffff8802265d30b8 Call Trace: [<ffffffff81103c56>] ? __perf_event_task_sched_out+0x36/0x50 [<ffffffff814dc84e>] __mutex_lock_slowpath+0x13e/0x180 [<ffffffff814dc6eb>] mutex_lock+0x2b/0x50 [<ffffffff8142b985>] rtnl_lock+0x15/0x20 [<ffffffffa04af17f>] bond_activebackup_arp_mon+0x24f/0x7b0 [bonding] [<ffffffffa04aef30>] ? bond_activebackup_arp_mon+0x0/0x7b0 [bonding] [<ffffffff81088830>] worker_thread+0x170/0x2a0 [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40 [<ffffffff810886c0>] ? worker_thread+0x0/0x2a0 [<ffffffff8108ddf6>] kthread+0x96/0xa0 [<ffffffff8100c1ca>] child_rip+0xa/0x20 [<ffffffff8108dd60>] ? kthread+0x0/0xa0 [<ffffffff8100c1c0>] ? child_rip+0x0/0x20 INFO: task modprobe:18386 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. modprobe D 0000000000000008 0 18386 18312 0x00000080 ffff88041b1e3c28 0000000000000086 0000000000000000 ffff88041b1e3c18 ffffffffa03b5e40 ffff880233379dc0 ffff88041b1e3c08 0000000100039845 ffff880433d050b8 ffff88041b1e3fd8 000000000000f598 ffff880433d050b8 Call Trace: [<ffffffffa03b5e40>] ? fib6_ifdown+0x0/0x30 [ipv6] [<ffffffff814dc065>] schedule_timeout+0x215/0x2e0 [<ffffffff814dbce3>] wait_for_common+0x123/0x180 [<ffffffff8105dc60>] ? default_wake_function+0x0/0x20 [<ffffffff814dbdfd>] wait_for_completion+0x1d/0x20 [<ffffffff81088c01>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81088cb0>] ? wq_barrier_func+0x0/0x20 [<ffffffff81088c58>] cleanup_workqueue_thread+0x28/0x80 [<ffffffff81088d73>] destroy_workqueue+0x73/0xc0 [<ffffffffa04acb8b>] bond_uninit+0x3b/0xe0 [bonding] [<ffffffff8141f66c>] rollback_registered+0x9c/0x130 [<ffffffff8141f722>] unregister_netdevice+0x22/0x70 [<ffffffffa04ad893>] bond_free_all+0x53/0xb0 [bonding] [<ffffffffa04b9d5d>] bonding_exit+0x35/0x3c [bonding] [<ffffffff810a9ad4>] sys_delete_module+0x194/0x260 [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0 [<ffffffff814e0c3e>] ? do_page_fault+0x3e/0xa0 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b Expected results: No hung. Additional info: System cannot reboot.
already fixed upstream with commit: 9e2e61fbf8ad016d24e4af0afff13505f3dd2a2a bonding: fix potential deadlock in bond_uninit()
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Patch(es) available on kernel-2.6.32-188.el6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2011-1530.html