Description of problem: GFS2 has at least one of the same deadlock issues as GFS1. Just did a quick test run and immediately got the folllowing read-write deadlock: PID: 28567 TASK: d2b31aa0 CPU: 0 COMMAND: "verify-data" #0 [d4a61cdc] schedule at c060ed00 #1 [d4a61d44] __mutex_lock_slowpath at c060f774 #2 [d4a61d60] .text.lock.mutex (via mutex_lock) at c060f7b2 #3 [d4a61d6c] __blockdev_direct_IO at c048ee57 #4 [d4a61e70] gfs2_direct_IO at f90927af #5 [d4a61ee4] pty_write at c0531efc #6 [d4a61efc] generic_file_read at c045526d #7 [d4a61f88] vfs_read at c046f09e #8 [d4a61fa4] sys_pread64 at c046f5dc #9 [d4a61fb8] system_call at c0403fc4 PID: 28561 TASK: d8438000 CPU: 1 COMMAND: "verify-data" #0 [dc82dc60] schedule at c060ed00 #1 [dc82dcc8] wait_for_completion at c060ee58 #2 [dc82dcf0] glock_wait_internal at f908a740 #3 [dc82dd14] gfs2_glock_nq at f908a9d7 #4 [dc82dd3c] gfs2_glock_nq_atime at f908bed7 #5 [dc82dd84] gfs2_prepare_write at f90933f9 #6 [dc82ddb8] generic_file_buffered_write at c045436b #7 [dc82ded4] __generic_file_write_nolock at c0454dc8 #8 [dc82df58] generic_file_write at c0454ebb #9 [dc82df88] vfs_write at c046ef47 #10 [dc82dfa4] sys_pwrite64 at c046f63b #11 [dc82dfb8] system_call at c0403fc4 Version-Release number of selected component (if applicable): How reproducible: Each time and every time Steps to Reproduce: just try to do direct read and write at the very same node under lock_dlm. Actual results: Expected results: Additional info:
Devel ACK, not setting blocker flag at this point but should fix before GA.
This looks odds - i_mutex is taken in generic_file_write, followed by glock while gfs2_direct_IO obtains i_mutex before glock. Nothing wrong with that. Then what is the __blockdev_direct_IO waiting for ? I'll re-run the test.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux major release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Major release. This request is not yet committed for inclusion.
The deadlock comes from: 1. Read side (i_mutex->glock->i_mutex): GFS2 uses XFS' DIO_OWN_LOCKING which will release i_mutex in the middle of __blockdev_direct_IO call after obtaining the SHARED mode glock upon entering gfs2_direct_IO: 1257 1258 if (dio_lock_type == DIO_OWN_LOCKING) { 1259 mutex_unlock(&inode->i_mutex); 1260 acquire_i_mutex = 1; 1261 } Then tries to obtain it before going out: 1284 out: 1285 if (release_i_mutex) 1286 mutex_unlock(&inode->i_mutex); 1287 else if (acquire_i_mutex) 1288 mutex_lock(&inode->i_mutex); 1289 return retval; 1290 } 1291 EXPORT_SYMBOL(__blockdev_direct_IO); 2. Write side (i_mutex->glock) GFS2 direct write obtains the i_mutex (after reader temporarily release i_mutex in the middle of the __blockdev_direct_IO) and then tries to acquire EXCLUSIVE glock and gets blocked due to reader's SHARED glock. So the headache here is that the gfs2 has been written around DIO_OWN_LOCKING logic so the tricks used in GFS1 CAN NOT apply here.
Created attachment 143748 [details] Patch to fix the hang This is very similar to Wendy's patch except for the alterations to the logic determining when DIO is skipped. This should allow us to build on the possibility of allocating writes for DIO in the future more easily. I've been doing some testing with verify-data and it seems that this fixes the problem.
184896
ignore previous useless comment in 2.6.18-1.2910.el5
A package has been built which should help the problem described in this bug report. This report is therefore being closed with a resolution of CURRENTRELEASE. You may reopen this bug report if the solution does not work for you.