Description of problem: Found an unlink deadlock where node A is removing a file in: PID: 6722 TASK: ffff81006c519820 CPU: 1 COMMAND: "rm" #0 [ffff81005a529ba8] schedule at ffffffff80060f39 #1 [ffff81005a529c90] just_schedule at ffffffff88433a74 #2 [ffff81005a529ca0] __wait_on_bit at ffffffff800619b4 #3 [ffff81005a529ce0] out_of_line_wait_on_bit at ffffffff80061a4f #4 [ffff81005a529d20] __cond_resched at ffffffff800899f3 #5 [ffff81005a529d50] glock_wait_internal at ffffffff88434f27 #6 [ffff81005a529d90] gfs2_glock_nq at ffffffff88435206 #7 [ffff81005a529dd0] gfs2_getattr at ffffffff8843fe89 #8 [ffff81005a529e08] gfs2_getattr at ffffffff8843fe81 #9 [ffff81005a529e10] vfs_getattr at ffffffff8000ddf5 #10 [ffff81005a529e40] vfs_lstat_fd at ffffffff8003e66a #11 [ffff81005a529ec0] do_unlinkat at ffffffff8003bcca #12 [ffff81005a529ef0] sys_newlstat at ffffffff8002a664 #13 [ffff81005a529f50] tracesys at ffffffff8005b229 #14 [ffff81005a529f80] tracesys at ffffffff8005b28d that is waiting for iopen lock. The lock_dlm1 in node B is demoting the iopen lock but, unfortunately, it is also tries to delete the inode at the same time. It is then blocked waiting on RGRP locks that seems to be only handled by itself ? PID: 3112 TASK: dad79550 CPU: 0 COMMAND: "lock_dlm1" #0 [cc1d9c98] schedule at c0603f79 #1 [cc1d9d00] just_schedule at e0410c52 #2 [cc1d9d04] __wait_on_bit at c06047ed #3 [cc1d9d1c] out_of_line_wait_on_bit at c0604872 #4 [cc1d9d54] wait_on_holder at e0410c49 #5 [cc1d9d64] glock_wait_internal at e0411db0 #6 [cc1d9d80] gfs2_glock_nq at e0412025 #7 [cc1d9dc4] do_strip at e04099f6 #8 [cc1d9e24] recursive_scan at e04087c9 #9 [cc1d9e58] recursive_scan at e0408818 #10 [cc1d9e94] recursive_scan at e0408818 #11 [cc1d9ed0] trunc_dealloc at e0408903 #12 [cc1d9f20] gfs2_delete_inode at e041ddac #13 [cc1d9f58] generic_delete_inode at c04853b7 #14 [cc1d9f64] iput at c0484e5b #15 [cc1d9f6c] blocking_cb at e0411366 #16 [cc1d9fcc] kthread at c0435f57 #17 [cc1d9fe4] kernel_thread_helper at c0405c39 Version-Release number of selected component (if applicable): How reproducible: Very likely Steps to Reproduce: Haven't sorted out the exact sequence yet. But I was doing sparse file read/write testing on two nodes with one single file using verify-data program. Frequently removed the file from either nodes and restart the job over and over again. Actual results: The "rm fileName" will hang forever. Expected results: No hang. Additional info:
Created attachment 192951 [details] node B glock dump
Created attachment 192961 [details] Node A glock dump
Will try to move this piece of logic into gfs2_glockd daemon.
This could end up hanging the whole cluster due to the blocked lock_dlm1.
Created attachment 193211 [details] RHEL5 patch Also add into overnight dd_io run to make sure it doesn't break anything else.
I don't think this patch is quite right... the newly added "from_deamon" argument to the function seems to be a duplicate with the "remote" argument, and due to this the branch: + if (from_daemon) { + gfs2_glock_schedule_for_reclaim(gl); + spin_unlock(&gl->gl_spin); + } else { will always be executed, so that the other branch is now never used. If this does the trick though, then I'm happy with making the substitution.
Yes, you're right. It was a quick and dirty patch. I didn't pay attention to that "remote" argument :). It is a duplicate. But don't pull into git tree yet. Would like to check something else. On the other hand, apparently dd_io is happy with the patch. Last night's loop also ran thru withouut issues.
Created attachment 193371 [details] RHEL5 revised patch
Posted to rh-kernel list. Moved to POST state.
in 2.6.18-48.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0959.html