Bug 173912 - GFS deadlock - gfs_write (do_write_direct) and gfs_setattr (do_truncate)
GFS deadlock - gfs_write (do_write_direct) and gfs_setattr (do_truncate)
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.0
All Linux
medium Severity high
: ---
: ---
Assigned To: Wendy Cheng
GFS Bugs
:
Depends On:
Blocks: 168429 175195
  Show dependency treegraph
 
Reported: 2005-11-22 10:27 EST by Wendy Cheng
Modified: 2007-11-30 17:07 EST (History)
0 users

See Also:
Fixed In Version: RHSA-2006-0132
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-03-07 15:52:53 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
gfs_kernel_i_alloc.patch (2.27 KB, patch)
2005-12-02 14:50 EST, Wendy Cheng
no flags Details | Diff

  None (edit)
Description Wendy Cheng 2005-11-22 10:27:36 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.7) Gecko/20050427 Red Hat/1.7.7-1.1.3.4

Description of problem:
GFS deadlock due to the following lock sequence:

(vfs) do_truncate {
        down(.... i_sem);
        down_write(.... i_alloc_sem);
        err = notify_change(dentry, &newattrs); -> call gfs_setattr -> glock
        up_write(.... i_alloc_sem);
        up(.... i_sem);
      }

(gfs) do_write_direct {
         glock
         down(.... i_sem)
         __blockdev_direct_IO -> down(.... i_alloc_sem) -> up(.... i_alloc_sem)
         up(.... i_sem)
       }
(gfs) gfs_read {
         glock
         _blockdev_direct_IO -> down(.... i_sem) -> up(.... i_sem)
       }
         



Version-Release number of selected component (if applicable):
2.6.9-22.ELsmp

How reproducible:
Sometimes

Steps to Reproduce:
1. (Running Oracle performance stress test)
2. Havn't tried using simpler test case.
  

Actual Results:  Oracle process hang. 

Additional info:
Comment 1 Wendy Cheng 2005-11-28 11:45:32 EST
Bugzilla 173912 - two GFS deadlocks during Oracle performance runs due to lock
order enforced by VFS layer where:
                                                                              
1) In sys_ftruncate()->do_truncate(), VFS layer grabs
  * i_sem
  * then i_alloc_sem
  * then call filesystem's setattr().
                                                                              
2) In Direct IO read patch, VFS layer calls
  * filesystem's direct_IO()
  * then grabs i_sem
  * followed by i_alloc_sem.
                                                                              
The above lock sequence would undoubtly cause deadlocks for external kernel
module that requires extra locking for various purposes. In GFS case, both
gfs_setattr() and gfs_direct_IO() need its own (global) locks to synchronize
inter-nodes (and inter-processes) control structures access. Two deadlocks
(bugzilla 171488, 173913 - bugzilla 173912 is for base kernel build) have been
found so far.

To work around this issue, a test kernel which forces GFS to take the two inode
semaphores in the earlier read code path was sent out before Thanksgiving break.
It reported back with successful results. The new RPMs require a base kernel
change (that allows GFS to bypass the inode semaphore acquiring within
__blockdev_direct_IO) - it basically does the following:

(vfs) sys_ftruncate()->do_truncate() {
       down(i_sem);
       down_write(i_alloc_sem);
       notify_change();
                   -> call gfs_setattr -> g-lock ->g-unlock
       up_write(i_alloc_sem);
       up(i_sem);
     }
(gfs) gfs_write (O_DIRECT) {
        down(i_sem)
        down(i_alloc_sem)
        g-lock
        __blockdev_direct_IO ->
                   new DIO_GFS_LOCKING {
                           don't get the inode semaphores }
        g-unlock
        up(i_alloc_sem)
        up(i_sem)
      }
(gfs) gfs_read (O_DIRECT) {
        down(i_sem)
        down(i_alloc_sem)
        g-lock
        _blockdev_direct_IO ->
                  new DIO_GFS_LOCKING {
                          don't get the inode sempahore }
        g-unlock
        up(i_alloc_sem)
        up(i_sem)
      }
                                                                               
                                                                    Would
appreciate if this patch could make into U3 - comment ? 
Comment 2 Wendy Cheng 2005-11-29 16:22:18 EST
This bugzilla was opened with an expectation that we needed to change base
kernel to workaround the above deadlock. However, after further investigation,
we've found if GFS switches from DIO_LOCKING to DIO_NO_LOCKING (that was
currently used by raw device), we can workaround the deadlock.

So I'm closing this as an non-bug. The GFS changes will use bugzilla 173913. 
Comment 3 Wendy Cheng 2005-12-02 14:47:39 EST
Unfortunately, we have to come back to this issue. We can't work around with
DIO_NO_LOCKING. The request has been sent to kernel group for review. Waiting
for the response. 
Comment 4 Wendy Cheng 2005-12-02 14:50:04 EST
Created attachment 121777 [details]
gfs_kernel_i_alloc.patch

New patch submitted to kernel group on 12/02/05.
Comment 9 Red Hat Bugzilla 2006-03-07 15:52:53 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2006-0132.html

Note You need to log in before you can comment on or make changes to this bug.