Bug 286821

Summary:

GFS2 lock_dlm1 deadlock itself

Product:

Red Hat Enterprise Linux 5

Reporter:

Wendy Cheng <nobody+wcheng>

Component:

kernel

Assignee:

Don Zickus <dzickus>

Status:

CLOSED ERRATA

QA Contact:

GFS Bugs <gfs-bugs>

Severity:

high

Docs Contact:

Priority:

high

Version:

5.1

CC:

lwang, rkenna

Target Milestone:

---

Target Release:

---

Hardware:

All

OS:

Linux

Whiteboard:

Fixed In Version:

RHBA-2007-0959

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2007-11-07 20:04:00 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
node B glock dump	none
Node A glock dump	none
RHEL5 patch	none
RHEL5 revised patch	none

Description Wendy Cheng 2007-09-11 20:08:13 UTC

Description of problem:

Found an unlink deadlock where node A is removing a file in:

PID: 6722   TASK: ffff81006c519820  CPU: 1   COMMAND: "rm"
 #0 [ffff81005a529ba8] schedule at ffffffff80060f39
 #1 [ffff81005a529c90] just_schedule at ffffffff88433a74
 #2 [ffff81005a529ca0] __wait_on_bit at ffffffff800619b4
 #3 [ffff81005a529ce0] out_of_line_wait_on_bit at ffffffff80061a4f
 #4 [ffff81005a529d20] __cond_resched at ffffffff800899f3
 #5 [ffff81005a529d50] glock_wait_internal at ffffffff88434f27
 #6 [ffff81005a529d90] gfs2_glock_nq at ffffffff88435206
 #7 [ffff81005a529dd0] gfs2_getattr at ffffffff8843fe89
 #8 [ffff81005a529e08] gfs2_getattr at ffffffff8843fe81
 #9 [ffff81005a529e10] vfs_getattr at ffffffff8000ddf5
#10 [ffff81005a529e40] vfs_lstat_fd at ffffffff8003e66a
#11 [ffff81005a529ec0] do_unlinkat at ffffffff8003bcca
#12 [ffff81005a529ef0] sys_newlstat at ffffffff8002a664
#13 [ffff81005a529f50] tracesys at ffffffff8005b229
#14 [ffff81005a529f80] tracesys at ffffffff8005b28d

that is waiting for iopen lock. The lock_dlm1 in node B is demoting
the iopen lock but, unfortunately, it is also tries to delete the 
inode at the same time. It is then blocked waiting on RGRP locks that
seems to be only handled by itself ?   

PID: 3112   TASK: dad79550  CPU: 0   COMMAND: "lock_dlm1"
 #0 [cc1d9c98] schedule at c0603f79
 #1 [cc1d9d00] just_schedule at e0410c52
 #2 [cc1d9d04] __wait_on_bit at c06047ed
 #3 [cc1d9d1c] out_of_line_wait_on_bit at c0604872
 #4 [cc1d9d54] wait_on_holder at e0410c49
 #5 [cc1d9d64] glock_wait_internal at e0411db0
 #6 [cc1d9d80] gfs2_glock_nq at e0412025
 #7 [cc1d9dc4] do_strip at e04099f6
 #8 [cc1d9e24] recursive_scan at e04087c9
 #9 [cc1d9e58] recursive_scan at e0408818
#10 [cc1d9e94] recursive_scan at e0408818
#11 [cc1d9ed0] trunc_dealloc at e0408903
#12 [cc1d9f20] gfs2_delete_inode at e041ddac
#13 [cc1d9f58] generic_delete_inode at c04853b7
#14 [cc1d9f64] iput at c0484e5b
#15 [cc1d9f6c] blocking_cb at e0411366
#16 [cc1d9fcc] kthread at c0435f57
#17 [cc1d9fe4] kernel_thread_helper at c0405c39

Version-Release number of selected component (if applicable):


How reproducible:
Very likely

Steps to Reproduce:
Haven't sorted out the exact sequence yet. But I was doing sparse
file read/write testing on two nodes with one single file using 
verify-data program. Frequently removed the file from either nodes 
and restart the job over and over again. 
  
Actual results:
The "rm fileName" will hang forever.

Expected results:
No hang.

Additional info:

Comment 1 Wendy Cheng 2007-09-11 20:08:13 UTC

Created attachment 192951 [details]
node B glock dump

Comment 2 Wendy Cheng 2007-09-11 20:11:33 UTC

Created attachment 192961 [details]
Node A glock dump

Comment 3 Wendy Cheng 2007-09-11 20:20:08 UTC

Will try to move this piece of logic into gfs2_glockd daemon.

Comment 4 Wendy Cheng 2007-09-11 20:32:04 UTC

This could end up hanging the whole cluster due to the blocked lock_dlm1.

Comment 6 Wendy Cheng 2007-09-12 05:01:18 UTC

Created attachment 193211 [details]
RHEL5 patch

Also add into overnight dd_io run to make sure it doesn't break anything else.

Comment 7 Steve Whitehouse 2007-09-12 09:21:13 UTC

I don't think this patch is quite right... the newly added "from_deamon"
argument to the function seems to be a duplicate with the "remote" argument, and
due to this the branch:

+			if (from_daemon) {
+				gfs2_glock_schedule_for_reclaim(gl);
+				spin_unlock(&gl->gl_spin);
+			} else { 

will always be executed, so that the other branch is now never used. If this
does the trick though, then I'm happy with making the substitution.

Comment 8 Wendy Cheng 2007-09-12 12:26:53 UTC

Yes, you're right. It was a quick and dirty patch. I didn't pay attention 
to that "remote" argument :). It is a duplicate. 

But don't pull into git tree yet. Would like to check something else.

On the other hand, apparently dd_io is happy with the patch. Last night's 
loop also ran thru withouut issues.

Comment 9 Wendy Cheng 2007-09-12 12:55:23 UTC

Created attachment 193371 [details]
RHEL5 revised patch

Comment 10 Wendy Cheng 2007-09-14 14:13:35 UTC

Posted to rh-kernel list. Moved to POST state.

Comment 11 Don Zickus 2007-09-18 19:23:39 UTC

in 2.6.18-48.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Comment 14 errata-xmlrpc 2007-11-07 20:04:00 UTC

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0959.html