Description of problem: Gluster commands hang after gluster volume stop. Please see dmesg output in Additional info. This was the internal Samba mount attempted to stop by the CTDB teardown hook. Actual results: Gluster volume stop does not succeed. Expected results: Would expect the gluster volume to stop successfully. Additional info: INFO: task umount:8609 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. umount D 0000000000000000 0 8609 8604 0x00000000 ffff88003c023b98 0000000000000086 0000000000000000 ffff88003c7f80c0 ffff88003c7f80f8 0000000000000000 ffff88003c023b58 ffffffff81060680 ffff88003c7f8678 ffff88003c023fd8 000000000000f4e8 ffff88003c7f8678 Call Trace: [<ffffffff81060680>] ? pick_next_task_fair+0xd0/0x130 [<ffffffff814ed3b5>] ? thread_return+0x1b3/0x76e [<ffffffff814ee0c5>] schedule_timeout+0x215/0x2e0 [<ffffffff8106266b>] ? enqueue_rt_entity+0x6b/0x80 [<ffffffff814edd43>] wait_for_common+0x123/0x180 [<ffffffff8105ea30>] ? default_wake_function+0x0/0x20 [<ffffffff814ede5d>] wait_for_completion+0x1d/0x20 [<ffffffff81060c21>] synchronize_sched_expedited+0x1c1/0x280 [<ffffffff810de0ae>] synchronize_rcu_expedited+0xe/0x10 [<ffffffff8113451a>] bdi_remove_from_list+0x4a/0x60 [<ffffffff811345e1>] bdi_unregister+0xb1/0x170 [<ffffffff81134796>] bdi_destroy+0xf6/0x150 [<ffffffffa031caa3>] fuse_conn_kill+0xc3/0xd0 [fuse] [<ffffffffa031cb1a>] fuse_put_super+0x6a/0x80 [fuse] [<ffffffff81178c8b>] generic_shutdown_super+0x5b/0xe0 [<ffffffff81178d76>] kill_anon_super+0x16/0x60 [<ffffffffa031bd92>] fuse_kill_sb_anon+0x52/0x60 [fuse] [<ffffffff81179d00>] deactivate_super+0x70/0x90 [<ffffffff81195caf>] mntput_no_expire+0xbf/0x110 [<ffffffff8119674b>] sys_umount+0x7b/0x3a0 [<ffffffff81081841>] ? sigprocmask+0x71/0x110 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b INFO: task umount:8609 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. umount D 0000000000000000 0 8609 8604 0x00000000 ffff88003c023b98 0000000000000086 0000000000000000 ffff88003c7f80c0 ffff88003c7f80f8 0000000000000000 ffff88003c023b58 ffffffff81060680 ffff88003c7f8678 ffff88003c023fd8 000000000000f4e8 ffff88003c7f8678 Call Trace: [<ffffffff81060680>] ? pick_next_task_fair+0xd0/0x130 [<ffffffff814ed3b5>] ? thread_return+0x1b3/0x76e [<ffffffff814ee0c5>] schedule_timeout+0x215/0x2e0 [<ffffffff8106266b>] ? enqueue_rt_entity+0x6b/0x80 [<ffffffff814edd43>] wait_for_common+0x123/0x180 [<ffffffff8105ea30>] ? default_wake_function+0x0/0x20 [<ffffffff814ede5d>] wait_for_completion+0x1d/0x20 [<ffffffff81060c21>] synchronize_sched_expedited+0x1c1/0x280 [<ffffffff810de0ae>] synchronize_rcu_expedited+0xe/0x10 [<ffffffff8113451a>] bdi_remove_from_list+0x4a/0x60 [<ffffffff811345e1>] bdi_unregister+0xb1/0x170 [<ffffffff81134796>] bdi_destroy+0xf6/0x150 [<ffffffffa031caa3>] fuse_conn_kill+0xc3/0xd0 [fuse] [<ffffffffa031cb1a>] fuse_put_super+0x6a/0x80 [fuse] [<ffffffff81178c8b>] generic_shutdown_super+0x5b/0xe0 [<ffffffff81178d76>] kill_anon_super+0x16/0x60 [<ffffffffa031bd92>] fuse_kill_sb_anon+0x52/0x60 [fuse] [<ffffffff81179d00>] deactivate_super+0x70/0x90 [<ffffffff81195caf>] mntput_no_expire+0xbf/0x110 [<ffffffff8119674b>] sys_umount+0x7b/0x3a0 [<ffffffff81081841>] ? sigprocmask+0x71/0x110 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b INFO: task umount:8609 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. umount D 0000000000000000 0 8609 8604 0x00000000 ffff88003c023b98 0000000000000086 0000000000000000 ffff88003c7f80c0 ffff88003c7f80f8 0000000000000000 ffff88003c023b58 ffffffff81060680 ffff88003c7f8678 ffff88003c023fd8 000000000000f4e8 ffff88003c7f8678 Call Trace: [<ffffffff81060680>] ? pick_next_task_fair+0xd0/0x130 [<ffffffff814ed3b5>] ? thread_return+0x1b3/0x76e [<ffffffff814ee0c5>] schedule_timeout+0x215/0x2e0 [<ffffffff8106266b>] ? enqueue_rt_entity+0x6b/0x80 [<ffffffff814edd43>] wait_for_common+0x123/0x180 [<ffffffff8105ea30>] ? default_wake_function+0x0/0x20 [<ffffffff814ede5d>] wait_for_completion+0x1d/0x20 [<ffffffff81060c21>] synchronize_sched_expedited+0x1c1/0x280 [<ffffffff810de0ae>] synchronize_rcu_expedited+0xe/0x10 [<ffffffff8113451a>] bdi_remove_from_list+0x4a/0x60 [<ffffffff811345e1>] bdi_unregister+0xb1/0x170 [<ffffffff81134796>] bdi_destroy+0xf6/0x150 [<ffffffffa031caa3>] fuse_conn_kill+0xc3/0xd0 [fuse] [<ffffffffa031cb1a>] fuse_put_super+0x6a/0x80 [fuse] [<ffffffff81178c8b>] generic_shutdown_super+0x5b/0xe0 [<ffffffff81178d76>] kill_anon_super+0x16/0x60 [<ffffffffa031bd92>] fuse_kill_sb_anon+0x52/0x60 [fuse] [<ffffffff81179d00>] deactivate_super+0x70/0x90 [<ffffffff81195caf>] mntput_no_expire+0xbf/0x110 [<ffffffff8119674b>] sys_umount+0x7b/0x3a0 [<ffffffff81081841>] ? sigprocmask+0x71/0x110 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b INFO: task umount:8609 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. umount D 0000000000000000 0 8609 8604 0x00000000 ffff88003c023b98 0000000000000086 0000000000000000 ffff88003c7f80c0 ffff88003c7f80f8 0000000000000000 ffff88003c023b58 ffffffff81060680 ffff88003c7f8678 ffff88003c023fd8 000000000000f4e8 ffff88003c7f8678 Call Trace: [<ffffffff81060680>] ? pick_next_task_fair+0xd0/0x130 [<ffffffff814ed3b5>] ? thread_return+0x1b3/0x76e [<ffffffff814ee0c5>] schedule_timeout+0x215/0x2e0 [<ffffffff8106266b>] ? enqueue_rt_entity+0x6b/0x80 [<ffffffff814edd43>] wait_for_common+0x123/0x180 [<ffffffff8105ea30>] ? default_wake_function+0x0/0x20 [<ffffffff814ede5d>] wait_for_completion+0x1d/0x20 [<ffffffff81060c21>] synchronize_sched_expedited+0x1c1/0x280 [<ffffffff810de0ae>] synchronize_rcu_expedited+0xe/0x10 [<ffffffff8113451a>] bdi_remove_from_list+0x4a/0x60 [<ffffffff811345e1>] bdi_unregister+0xb1/0x170 [<ffffffff81134796>] bdi_destroy+0xf6/0x150 [<ffffffffa031caa3>] fuse_conn_kill+0xc3/0xd0 [fuse] [<ffffffffa031cb1a>] fuse_put_super+0x6a/0x80 [fuse] [<ffffffff81178c8b>] generic_shutdown_super+0x5b/0xe0 [<ffffffff81178d76>] kill_anon_super+0x16/0x60 [<ffffffffa031bd92>] fuse_kill_sb_anon+0x52/0x60 [fuse] [<ffffffff81179d00>] deactivate_super+0x70/0x90 [<ffffffff81195caf>] mntput_no_expire+0xbf/0x110 [<ffffffff8119674b>] sys_umount+0x7b/0x3a0 [<ffffffff81081841>] ? sigprocmask+0x71/0x110 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Csaba, can you take a look at this?
This bug is not seen in current master branch (which will get branched as RHS 2.1.0 soon). To consider it for fixing, want to make sure this bug still exists in RHS servers. If not reproduced, would like to close this.
For QE: there was no particular fix which went into fix the issue, could be by sideeffect of other patches. Please see if initial steps reported in Description work fine now.
BUG verified on version: glusterfs 3.4.0.12rhs.beta1 built on Jun 28 2013 06:41:37
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html