Bug 212627 - GFS2 Direct IO deadlock
GFS2 Direct IO deadlock
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Whitehouse
GFS Bugs
:
Depends On:
Blocks: 218883
  Show dependency treegraph
 
Reported: 2006-10-27 15:20 EDT by Wendy Cheng
Modified: 2007-11-30 17:07 EST (History)
4 users (show)

See Also:
Fixed In Version: beta2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-12-22 20:48:55 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dio deadlock fix (1.17 KB, patch)
2006-10-31 11:02 EST, Wendy Cheng
no flags Details | Diff
Patch to fix the hang (2.85 KB, patch)
2006-12-15 06:52 EST, Steve Whitehouse
no flags Details | Diff

  None (edit)
Description Wendy Cheng 2006-10-27 15:20:51 EDT
Description of problem:
GFS2 has at least one of the same deadlock issues as GFS1. Just did a quick
test run and immediately got the folllowing read-write deadlock:

PID: 28567  TASK: d2b31aa0  CPU: 0   COMMAND: "verify-data"
 #0 [d4a61cdc] schedule at c060ed00
 #1 [d4a61d44] __mutex_lock_slowpath at c060f774
 #2 [d4a61d60] .text.lock.mutex (via mutex_lock) at c060f7b2
 #3 [d4a61d6c] __blockdev_direct_IO at c048ee57
 #4 [d4a61e70] gfs2_direct_IO at f90927af
 #5 [d4a61ee4] pty_write at c0531efc
 #6 [d4a61efc] generic_file_read at c045526d
 #7 [d4a61f88] vfs_read at c046f09e
 #8 [d4a61fa4] sys_pread64 at c046f5dc
 #9 [d4a61fb8] system_call at c0403fc4

PID: 28561  TASK: d8438000  CPU: 1   COMMAND: "verify-data"
 #0 [dc82dc60] schedule at c060ed00
 #1 [dc82dcc8] wait_for_completion at c060ee58
 #2 [dc82dcf0] glock_wait_internal at f908a740
 #3 [dc82dd14] gfs2_glock_nq at f908a9d7
 #4 [dc82dd3c] gfs2_glock_nq_atime at f908bed7
 #5 [dc82dd84] gfs2_prepare_write at f90933f9
 #6 [dc82ddb8] generic_file_buffered_write at c045436b
 #7 [dc82ded4] __generic_file_write_nolock at c0454dc8
 #8 [dc82df58] generic_file_write at c0454ebb
 #9 [dc82df88] vfs_write at c046ef47
#10 [dc82dfa4] sys_pwrite64 at c046f63b
#11 [dc82dfb8] system_call at c0403fc4

Version-Release number of selected component (if applicable):


How reproducible:
Each time and every time

Steps to Reproduce:
just try to do direct read and write at the very same node under lock_dlm.

Actual results:


Expected results:


Additional info:
Comment 1 Kiersten (Kerri) Anderson 2006-10-27 16:05:38 EDT
Devel ACK, not setting blocker flag at this point but should fix before GA.
Comment 2 Wendy Cheng 2006-10-27 16:15:27 EDT
This looks odds - i_mutex is taken in generic_file_write, followed by 
glock while gfs2_direct_IO obtains i_mutex before glock. Nothing wrong
with that. Then what is the __blockdev_direct_IO waiting for ?

I'll re-run the test.
Comment 3 RHEL Product and Program Management 2006-10-27 16:20:38 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 Wendy Cheng 2006-10-27 17:14:45 EDT
The deadlock comes from: 

1. Read side (i_mutex->glock->i_mutex):

   GFS2 uses XFS' DIO_OWN_LOCKING which will release i_mutex in the
   middle of __blockdev_direct_IO call after obtaining the SHARED 
   mode glock upon entering gfs2_direct_IO:

   1257
   1258                         if (dio_lock_type == DIO_OWN_LOCKING) {
   1259                                 mutex_unlock(&inode->i_mutex);
   1260                                 acquire_i_mutex = 1;
   1261                         }

   Then tries to obtain it before going out: 

   1284 out:
   1285         if (release_i_mutex)
   1286                 mutex_unlock(&inode->i_mutex);
   1287         else if (acquire_i_mutex)
   1288                 mutex_lock(&inode->i_mutex);
   1289         return retval;
   1290 }
   1291 EXPORT_SYMBOL(__blockdev_direct_IO);

2. Write side (i_mutex->glock)

   GFS2 direct write obtains the i_mutex (after reader temporarily 
   release i_mutex in the middle of the __blockdev_direct_IO) and then
   tries to acquire EXCLUSIVE glock and gets blocked due to reader's
   SHARED glock.
   
So the headache here is that the gfs2 has been written around DIO_OWN_LOCKING
logic so the tricks used in GFS1 CAN NOT apply here.
Comment 5 RHEL Product and Program Management 2006-10-27 17:20:46 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 8 RHEL Product and Program Management 2006-10-27 17:40:44 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 19 Steve Whitehouse 2006-12-15 06:52:47 EST
Created attachment 143748 [details]
Patch to fix the hang

This is very similar to Wendy's patch except for the alterations to the logic
determining when DIO is skipped. This should allow us to build on the
possibility of allocating writes for DIO in the future more easily.

I've been doing some testing with verify-data and it seems that this fixes the
problem.
Comment 20 Don Zickus 2006-12-17 21:52:56 EST
184896
Comment 21 Don Zickus 2006-12-18 12:42:14 EST
ignore previous useless comment
in 2.6.18-1.2910.el5
Comment 22 RHEL Product and Program Management 2006-12-22 20:48:56 EST
A package has been built which should help the problem described in 
this bug report. This report is therefore being closed with a resolution 
of CURRENTRELEASE. You may reopen this bug report if the solution does 
not work for you.

Note You need to log in before you can comment on or make changes to this bug.