Here is what I'm seeing:  link-13 and link-14 look like they're both
waiting for glocks that they already hold.  They both have "trylock"
holders for the file /mnt/gfs2/writev-read.  Here are the pertinent
lockdump sections from all nodes:

link-13:------------------------------------------------------------
Glock 0xe000000002585b48 (2, 0x220efe3)
  gl_flags = 1 
  gl_ref = 6
  gl_state = 0
  gl_owner = pid 6112 (d_doio)
  gl_ip = 11529215054675295824
  req_gh = yes
  lvb_count = 0
  object = yes
  le = no
  reclaim = no
  aspace = 0xe000000003dc8180 nrpages = 0
  ail = 0
  Request
    owner = 6112 (d_doio)
    gh_state = 3
    gh_flags = 1 9 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_readpage+0xc0/0x280 [gfs2]
  Waiter3
    owner = 6112 (d_doio)
    gh_state = 3
    gh_flags = 1 9 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_readpage+0xc0/0x280 [gfs2]
  Waiter3
    owner = 6133 (ls)
    gh_state = 3
    gh_flags = 3 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_getattr+0x150/0x240 [gfs2]
  Inode: busy

link-14:------------------------------------------------------------
Glock 0xe00000002ca17b98 (2, 0x220efe3)
  gl_flags = 1 
  gl_ref = 6
  gl_state = 0
  gl_owner = pid 5929 (d_doio)
  gl_ip = 11529215054675295824
  req_gh = yes
  lvb_count = 0
  object = yes
  le = no
  reclaim = no
  aspace = 0xe00000002ca26da0 nrpages = 0
  ail = 0
  Request
    owner = 5929 (d_doio)
    gh_state = 3
    gh_flags = 1 9 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_readpage+0xc0/0x280 [gfs2]
  Waiter3
    owner = 5929 (d_doio)
    gh_state = 3
    gh_flags = 1 9 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_readpage+0xc0/0x280 [gfs2]
  Waiter3
    owner = 5949 (ls)
    gh_state = 3
    gh_flags = 3 
    error = 0
    gh_iflags = 1 10 
    initialized at: gfs2_getattr+0x150/0x240 [gfs2]
  Inode: busy

link-15:------------------------------------------------------------
Glock 0xe000000028ffd858 (2, 0x220efe3)
  gl_flags = (unlocked) 
  gl_ref = 3
  gl_state = 3
  gl_owner = 5840 (ended)
  gl_ip = 11529215054675263056
  req_gh = no
  lvb_count = 0
  object = yes
  le = no
  reclaim = no
  aspace = 0xe00000002814c430 nrpages = 1
  ail = 0
  Inode:
    num = 25/35712995
    type = 8
    i_flags = 

link-16:------------------------------------------------------------
Glock 0xe0000040657e19d0 (2, 0x220efe3)
  gl_flags = (unlocked) 
  gl_ref = 3
  gl_state = 3
  gl_owner = 7416 (ended)
  gl_ip = 11529215054678753552
  req_gh = no
  req_bh = no
  lvb_count = 0
  object = yes
  le = no
  reclaim = no
  aspace = 0xe000004063480448 nrpages = 1
  ail = 0
  Inode:
    num = 25/35712995
    type = 8
    i_flags = 

Okay, Steve, here is my theory.  Does this make sense?

The glock is in the unlocked state, but it has a holder.  So it was
once locked and now isn't.  However, the holder didn't get taken off
the owner's list (gl_req_gh).  This is hard to explain, but my theory
is that this sequence of events might be occurring:

1. The glock is locked, so the holder is put on the gl_req_gh list.
2. Function drop_bh is called to drop the holder.
3. drop_bh initializes "*gh = gl->gl_req_gh;" without the gl_spin lock.
4. drop_bh calls state_change
5. drop_bh calls go_inval
6. In the mean time, a holder gets put onto the gl_req_bh list.
7. go_inval finishes and returns back to drop_bh
8. Now we've got a holder, but gh is still NULL, so it never deletes
   the holder off the list.

In other words, shouldn't function drop_bh() protect its setting of
gh = gl->gl_req_gh with the gl_spin held, and perhaps only do it
after the return from go_inval()?  Perhaps drop_bh should look more
like this:

	struct gfs2_holder *gh;
...
	state_change(gl, LM_ST_UNLOCKED);
	gl->gl_req_gh;

	if (glops->go_inval)
		glops->go_inval(gl, DIO_METADATA);
	spin_lock(&gl->gl_spin);
	gh = gl->gl_req_gh;
	if (gh) {
		list_del_init(&gh->gh_list);
		gh->gh_error = 0;
	}
	spin_unlock(&gl->gl_spin);
Since there's a spin_lock immediately after, we can eliminate the
unlock/relock from the code too.

I don't know if this theory holds any water...

Function xmote_bh does the same kind of thing.

gl_req_gh isn't a list, its a pointer to the lock request which caused the
current state change and is only valid during a demote or promote which involved
some remote operation.

These are potentially duplicates of bz 432370.  Marking NEEDINFO to have QE try
to recreate.

This looks exactly like #432057, so unless this can still be recreated then I'll
mark it as a dup of that bug.

*** This bug has been marked as a duplicate of 432057 ***