Bug 286821 - GFS2 lock_dlm1 deadlock itself
GFS2 lock_dlm1 deadlock itself
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
All Linux
high Severity high
: ---
: ---
Assigned To: Don Zickus
GFS Bugs
Depends On:
  Show dependency treegraph
Reported: 2007-09-11 16:08 EDT by Wendy Cheng
Modified: 2007-11-30 17:07 EST (History)
2 users (show)

See Also:
Fixed In Version: RHBA-2007-0959
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-11-07 15:04:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
node B glock dump (29.58 KB, text/plain)
2007-09-11 16:08 EDT, Wendy Cheng
no flags Details
Node A glock dump (30.57 KB, text/plain)
2007-09-11 16:11 EDT, Wendy Cheng
no flags Details
RHEL5 patch (2.17 KB, patch)
2007-09-12 01:01 EDT, Wendy Cheng
no flags Details | Diff
RHEL5 revised patch (564 bytes, patch)
2007-09-12 08:55 EDT, Wendy Cheng
no flags Details | Diff

  None (edit)
Description Wendy Cheng 2007-09-11 16:08:13 EDT
Description of problem:

Found an unlink deadlock where node A is removing a file in:

PID: 6722   TASK: ffff81006c519820  CPU: 1   COMMAND: "rm"
 #0 [ffff81005a529ba8] schedule at ffffffff80060f39
 #1 [ffff81005a529c90] just_schedule at ffffffff88433a74
 #2 [ffff81005a529ca0] __wait_on_bit at ffffffff800619b4
 #3 [ffff81005a529ce0] out_of_line_wait_on_bit at ffffffff80061a4f
 #4 [ffff81005a529d20] __cond_resched at ffffffff800899f3
 #5 [ffff81005a529d50] glock_wait_internal at ffffffff88434f27
 #6 [ffff81005a529d90] gfs2_glock_nq at ffffffff88435206
 #7 [ffff81005a529dd0] gfs2_getattr at ffffffff8843fe89
 #8 [ffff81005a529e08] gfs2_getattr at ffffffff8843fe81
 #9 [ffff81005a529e10] vfs_getattr at ffffffff8000ddf5
#10 [ffff81005a529e40] vfs_lstat_fd at ffffffff8003e66a
#11 [ffff81005a529ec0] do_unlinkat at ffffffff8003bcca
#12 [ffff81005a529ef0] sys_newlstat at ffffffff8002a664
#13 [ffff81005a529f50] tracesys at ffffffff8005b229
#14 [ffff81005a529f80] tracesys at ffffffff8005b28d

that is waiting for iopen lock. The lock_dlm1 in node B is demoting
the iopen lock but, unfortunately, it is also tries to delete the 
inode at the same time. It is then blocked waiting on RGRP locks that
seems to be only handled by itself ?   

PID: 3112   TASK: dad79550  CPU: 0   COMMAND: "lock_dlm1"
 #0 [cc1d9c98] schedule at c0603f79
 #1 [cc1d9d00] just_schedule at e0410c52
 #2 [cc1d9d04] __wait_on_bit at c06047ed
 #3 [cc1d9d1c] out_of_line_wait_on_bit at c0604872
 #4 [cc1d9d54] wait_on_holder at e0410c49
 #5 [cc1d9d64] glock_wait_internal at e0411db0
 #6 [cc1d9d80] gfs2_glock_nq at e0412025
 #7 [cc1d9dc4] do_strip at e04099f6
 #8 [cc1d9e24] recursive_scan at e04087c9
 #9 [cc1d9e58] recursive_scan at e0408818
#10 [cc1d9e94] recursive_scan at e0408818
#11 [cc1d9ed0] trunc_dealloc at e0408903
#12 [cc1d9f20] gfs2_delete_inode at e041ddac
#13 [cc1d9f58] generic_delete_inode at c04853b7
#14 [cc1d9f64] iput at c0484e5b
#15 [cc1d9f6c] blocking_cb at e0411366
#16 [cc1d9fcc] kthread at c0435f57
#17 [cc1d9fe4] kernel_thread_helper at c0405c39

Version-Release number of selected component (if applicable):

How reproducible:
Very likely

Steps to Reproduce:
Haven't sorted out the exact sequence yet. But I was doing sparse
file read/write testing on two nodes with one single file using 
verify-data program. Frequently removed the file from either nodes 
and restart the job over and over again. 
Actual results:
The "rm fileName" will hang forever.

Expected results:
No hang.

Additional info:
Comment 1 Wendy Cheng 2007-09-11 16:08:13 EDT
Created attachment 192951 [details]
node B glock dump
Comment 2 Wendy Cheng 2007-09-11 16:11:33 EDT
Created attachment 192961 [details]
Node A glock dump
Comment 3 Wendy Cheng 2007-09-11 16:20:08 EDT
Will try to move this piece of logic into gfs2_glockd daemon.  
Comment 4 Wendy Cheng 2007-09-11 16:32:04 EDT
This could end up hanging the whole cluster due to the blocked lock_dlm1. 
Comment 6 Wendy Cheng 2007-09-12 01:01:18 EDT
Created attachment 193211 [details]
RHEL5 patch

Also add into overnight dd_io run to make sure it doesn't break anything else.
Comment 7 Steve Whitehouse 2007-09-12 05:21:13 EDT
I don't think this patch is quite right... the newly added "from_deamon"
argument to the function seems to be a duplicate with the "remote" argument, and
due to this the branch:

+			if (from_daemon) {
+				gfs2_glock_schedule_for_reclaim(gl);
+				spin_unlock(&gl->gl_spin);
+			} else { 

will always be executed, so that the other branch is now never used. If this
does the trick though, then I'm happy with making the substitution.
Comment 8 Wendy Cheng 2007-09-12 08:26:53 EDT
Yes, you're right. It was a quick and dirty patch. I didn't pay attention 
to that "remote" argument :). It is a duplicate. 

But don't pull into git tree yet. Would like to check something else.

On the other hand, apparently dd_io is happy with the patch. Last night's 
loop also ran thru withouut issues. 

Comment 9 Wendy Cheng 2007-09-12 08:55:23 EDT
Created attachment 193371 [details]
RHEL5 revised patch
Comment 10 Wendy Cheng 2007-09-14 10:13:35 EDT
Posted to rh-kernel list. Moved to POST state. 
Comment 11 Don Zickus 2007-09-18 15:23:39 EDT
in 2.6.18-48.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 14 errata-xmlrpc 2007-11-07 15:04:00 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.