Bug 455309 - gfs umount deadlock dlm:release_lockspace
gfs umount deadlock dlm:release_lockspace
Status: CLOSED DUPLICATE of bug 488318
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: gfs (Show other bugs)
4
All Linux
high Severity high
: ---
: ---
Assigned To: Abhijith Das
Cluster QE
:
: 495969 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-07-14 15:21 EDT by Corey Marthaler
Modified: 2010-01-11 22:17 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-05-07 17:28:31 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log from grant-01 (114.72 KB, text/plain)
2008-07-14 15:26 EDT, Corey Marthaler
no flags Details
log from grant-02 (127.45 KB, text/plain)
2008-07-14 15:27 EDT, Corey Marthaler
no flags Details
log from grant-03 (111.92 KB, text/plain)
2008-07-14 15:28 EDT, Corey Marthaler
no flags Details
sosreport from node135 (1.93 MB, application/x-bzip2)
2009-04-17 09:51 EDT, Jason Willeford
no flags Details
Bob's Tech Notes (5.81 KB, text/plain)
2009-04-22 19:18 EDT, Robert Peterson
no flags Details

  None (edit)
Description Corey Marthaler 2008-07-14 15:21:33 EDT
Description of problem:
Hit this while running mount_stress during 4.7 regression testing. The test was
unmounting 1 of the 10 gfs filesystems when it deadlocked.

umount        D 0000000000000000     0 32290   2407                     (NOTLB)
000001010f087cf8 0000000000000002 0000010120008030 00000100d75a7240
00000100dfeba030 0000000000000001 00000100d75a7240 000000012005e7e0
00000100dfeba030 0000000000000e7b
Call Trace:
       <ffffffff80318548>{wait_for_completion+167}
       <ffffffff80134709>{default_wake_function+0}
       <ffffffff80134709>{default_wake_function+0}
       <ffffffff8014c870>{kthread_stop+147}
       <ffffffffa0284fe9>{:dlm:release_lockspace+248}
       <ffffffffa0319515>{:lock_dlm:release_gdlm+15}
       <ffffffffa0319bf3>{:lock_dlm:lm_dlm_unmount+54}
       <ffffffffa029f3c9>{:lock_harness:lm_unmount+62}
       <ffffffffa02baf8b>{:gfs:gfs_lm_unmount+33}
       <ffffffffa02ca318>{:gfs:gfs_put_super+806}
       <ffffffff801814e5>{generic_shutdown_super+198}
       <ffffffffa02c7a66>{:gfs:gfs_kill_sb+41}
       <ffffffff80181406>{deactivate_super+95}
       <ffffffff80197ab6>{sys_umount+925}
       <ffffffff801848eb>{sys_newstat+17}
       <ffffffff80110e1d>{error_exit+0}
       <ffffffff801102f6>{system_call+126}


[root@grant-01 ~]# cman_tool services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           2   2 run       -
[1 2 3]

DLM Lock Space:  "clvmd"                             3   3 run       -
[1 2 3]

DLM Lock Space:  "GRANT7"                          100  86 run       -
[1]

DLM Lock Space:  "GRANT0"                          102  88 run       -
[1]

GFS Mount Group: "GRANT7"                          101  87 run       -
[1]

GFS Mount Group: "GRANT0"                          103  89 run       -
[1]


[root@grant-02 ~]# cman_tool services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           2   2 run       -
[1 2 3]

DLM Lock Space:  "clvmd"                             3   3 run       -
[1 2 3]

DLM Lock Space:  "GRANT4"                           75  32 run       -
[2]

DLM Lock Space:  "GRANT8"                           79  34 run       -
[2]

DLM Lock Space:  "GRANT2"                           83  36 run       -
[2 3]

GFS Mount Group: "GRANT4"                           77  33 run       -
[2]

GFS Mount Group: "GRANT8"                           81  35 run       -
[2]

GFS Mount Group: "GRANT2"                           85  37 run       -
[2 3]


[root@grant-03 ~]# cman_tool services
Service          Name                              GID LID State     Code
Fence Domain:    "default"                           2   2 run       -
[1 2 3]

DLM Lock Space:  "clvmd"                             3   3 run       -
[1 2 3]

DLM Lock Space:  "GRANT2"                           83  62 run       -
[2 3]

GFS Mount Group: "GRANT2"                           85  63 run       -
[2 3]


Version-Release number of selected component (if applicable):
2.6.9-78.ELsmp
[root@grant-03 ~]# rpm -qa | grep GFS
GFS-6.1.18-1
GFS-kernel-2.6.9-80.9
GFS-kernel-smp-2.6.9-80.9
GFS-kernel-xenU-2.6.9-80.9
GFS-kernheaders-2.6.9-80.9
GFS-kernel-largesmp-2.6.9-80.9
GFS-debuginfo-6.1.18-1
GFS-kernel-debuginfo-2.6.9-80.9
Comment 1 Corey Marthaler 2008-07-14 15:26:45 EDT
Created attachment 311756 [details]
log from grant-01
Comment 2 Corey Marthaler 2008-07-14 15:27:02 EDT
Created attachment 311757 [details]
log from grant-02
Comment 3 Corey Marthaler 2008-07-14 15:28:09 EDT
Created attachment 311758 [details]
log from grant-03
Comment 4 Steve Whitehouse 2008-12-10 12:00:57 EST
Is this reproducable, or just a one off?
Comment 5 Abhijith Das 2009-03-30 17:23:15 EDT
No new information, moved to 4.9
Comment 6 Robert Peterson 2009-04-16 14:58:49 EDT
*** Bug 495969 has been marked as a duplicate of this bug. ***
Comment 11 Jason Willeford 2009-04-17 09:51:31 EDT
Created attachment 340011 [details]
sosreport from node135

Added sosreport from node135 from IT283633.
Comment 14 Issue Tracker 2009-04-20 15:05:09 EDT
That errata has already been applied to this server before the crash. Is
this  a new manfestation of the same problem or something new?


This event sent from IssueTracker by calvin_g_smith 
 issue 283633
Comment 18 Corey Marthaler 2009-04-22 16:31:37 EDT
QA is currently attempting to reproduce this issue on two different 4.8 clusters. The last time we saw it however was 9 months ago.
Comment 19 Robert Peterson 2009-04-22 19:18:34 EDT
Created attachment 340847 [details]
Bob's Tech Notes

I spent a couple hours analyzing the crash dump.  The unmount is
waiting to acquire the glock associated with syncing the statfs_fast
file.  The glock shows it is "held" by a process that has already
ended as far as I can tell.

The most important thing to note: This goes back to statfs_fast=1
as we've seen before lately.
Comment 20 Robert Peterson 2009-04-22 19:20:26 EDT
Adding myself and Dave T. to the cc list.  Bottom line: We're still
looking at the problem and trying to figure out what went wrong.
Hopefully my attachment from comment #19 will help.
Comment 21 Corey Marthaler 2009-04-23 11:08:17 EDT
FWIW - QA was unable to reproduce any problems running mount stress tests to 15 gfs on two different clusters.
Comment 22 Robert Peterson 2009-04-23 11:15:30 EDT
Hopefully you were using statfs_fast for your tests?
Comment 23 David Teigland 2009-04-23 15:37:40 EDT
First, it doesn't look like the stuck unmount in bug 495969 is the same as the original bug which is stuck on kthread_stop.  So, let's forget about everything prior to comment 6 and consider that bug closed, we'll redefine this bz to be the new issue.

Bob, I was surprised to see multiple holders on the linode glock, I only expected that the gfs_quotad thread would be taking that glock when syncing the statfs info.  Looking through the code, it appears that's not quite true.  If you run gfs_tool to set statfs_fast, gfs_tool will call gfs_statfs_init() -> gfs_statfs_start() which takes the glock.  It doesn't appear that df would ever take it, and I can't see any other possibilities.  It seems likely that the statfs_fast code is at fault somehow, and that's not used by QA AFAIK.

Can you look at the dlm lockspace to see what the state of that glock is?  You could also look at lock_dlm's copy.  The lock_dlm dlm_lock_t struct is accessible through gl->gl_lock in the crash dump.
Comment 24 Corey Marthaler 2009-04-23 16:12:32 EDT
Last night I wasn't using statfs_fast, but I hacked up the test to set it on every mount option so that it would be on during all the unmounting. I still haven't seen a unmount hang yet (it's been running for 3 hours now).

Let me know if there's something else I should be trying.
Comment 25 Robert Peterson 2009-04-23 17:23:20 EDT
struct dlm_lock 0x1009eb346c0
struct dlm_lock {
  dlm = 0x117fd14e200, 
  lockname = {
    ln_number = 25, 
    ln_type = 2
  }, 
  lvb = 0x0, 
  lksb = {
    sb_status = 0, 
    sb_lkid = 92930403, 
    sb_flags = 0 '\0', 
    sb_lvbptr = 0x0
  }, 
  cur = 5, 
  req = 5, 
  prev_req = 0, 
  lkf = 0, 
  type = 0, 
  flags = 0, 
  bast_mode = 0, 
  uast_wait = {
    done = 0, 
    wait = {
      lock = {
        lock = 1, 
        magic = 3735899821
      }, 
      task_list = {
        next = 0x1009eb34728, 
        prev = 0x1009eb34728
      }
    }
  }, 
  clist = {
    next = 0x100100, 
    prev = 0x200200
  }, 
  blist = {
    next = 0x100100, 
    prev = 0x200200
  }, 
  dlist = {
    next = 0x100100, 
    prev = 0x200200
  }, 
  slist = {
    next = 0x100100, 
    prev = 0x200200
  }, 
  hold_null = 0x0, 
  posix = 0x0, 
  null_list = {
    next = 0x0, 
    prev = 0x0
  }
}
Comment 26 David Teigland 2009-04-23 17:36:21 EDT
The following messages are the same issue as bug 495600:

dlm: gfs0: cancel reply ret 0
lock_dlm: unlock sb_status 0 2,19 flags 0
dlm: gfs0: process_lockqueue_reply id 58a0163 state 0
Comment 27 David Teigland 2009-04-24 12:21:16 EDT
Mar 30 06:18:29 -- dlm bug 495600 occured on the fast_statfs glock 2,0x19.
The messages in comment 26 show this.  A holder is left waiting for that
unlock to complete (which never will):

gl_req_gh = 0x11806b42180,
gl_req_bh = 0xffffffffa025255a <drop_bh>,
    
crash> struct gfs_holder 0x11806b42180
struct gfs_holder {
  gh_list = {
    next = 0x117f9a3f480, 
    prev = 0x117f9a3f480
  }, 
  gh_gl = 0x117f9a3f428,
  gh_owner = 0x0, 
  gh_state = 0,
  gh_flags = 1, 
  gh_error = 0,
  gh_iflags = 52,
  gh_wait = {
    done = 0, 
    wait = {
      lock = {
        lock = 1, 
        magic = 3735899821
      },
      task_list = {
        next = 0x11806b421c8,
        prev = 0x11806b421c8
      }
    }
  }
}

When the umount happens, the umount blocks waiting for gfs_quotad to exit, but gfs_quotad is blocked trying to get an EX lock on the fast_statfs glock which doesn't happen because the unlock above is not completing.

crash> struct gfs_holder 0x117f85f9e88
struct gfs_holder {
  gh_list = {
    next = 0x117f9a3f490,
    prev = 0x117f9a3f490
  },
  gh_gl = 0x117f9a3f428,
  gh_owner = 0x117fa22f7f0,  pid 14987 gfs_quotad
  gh_state = 1,
  gh_flags = 1056,
  gh_error = 0,
  gh_iflags = 2,
  gh_wait = {
    done = 0,
    wait = {
      lock = {
        lock = 1,
        magic = 3735899821
      },
      task_list = {
        next = 0x117f85f9da0,
        prev = 0x117f85f9da0
      }
    }
  }
}

I would think we'd see a process somewhere waiting for the unlock to complete, but I don't.  This is the fast_statfs glock we're talking about, though, so it could be used in some incorrect, unknown way, different from normal glocks.
Comment 30 David Teigland 2009-05-07 12:46:04 EDT
We're hoping that this is a duplicate of bug 495600.
We're also hoping that a new fast_statfs implementation might materialize.
Comment 31 Robert Peterson 2009-05-07 17:28:31 EDT
The patch for bug #488318 will fix this problem.  Is the customer
willing to try the patch for bug #488318?

*** This bug has been marked as a duplicate of bug 488318 ***

Note You need to log in before you can comment on or make changes to this bug.