Bug 173912

Summary: GFS deadlock - gfs_write (do_write_direct) and gfs_setattr (do_truncate)
Product: Red Hat Enterprise Linux 4 Reporter: Wendy Cheng <wcheng>
Component: kernelAssignee: Wendy Cheng <wcheng>
Status: CLOSED ERRATA QA Contact: GFS Bugs <gfs-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 4.0   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Fixed In Version: RHSA-2006-0132 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2006-03-07 20:52:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 168429, 175195    
Description Flags
gfs_kernel_i_alloc.patch none

Description Wendy Cheng 2005-11-22 15:27:36 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.7) Gecko/20050427 Red Hat/1.7.7-

Description of problem:
GFS deadlock due to the following lock sequence:

(vfs) do_truncate {
        down(.... i_sem);
        down_write(.... i_alloc_sem);
        err = notify_change(dentry, &newattrs); -> call gfs_setattr -> glock
        up_write(.... i_alloc_sem);
        up(.... i_sem);

(gfs) do_write_direct {
         down(.... i_sem)
         __blockdev_direct_IO -> down(.... i_alloc_sem) -> up(.... i_alloc_sem)
         up(.... i_sem)
(gfs) gfs_read {
         _blockdev_direct_IO -> down(.... i_sem) -> up(.... i_sem)

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. (Running Oracle performance stress test)
2. Havn't tried using simpler test case.

Actual Results:  Oracle process hang. 

Additional info:

Comment 1 Wendy Cheng 2005-11-28 16:45:32 UTC
Bugzilla 173912 - two GFS deadlocks during Oracle performance runs due to lock
order enforced by VFS layer where:
1) In sys_ftruncate()->do_truncate(), VFS layer grabs
  * i_sem
  * then i_alloc_sem
  * then call filesystem's setattr().
2) In Direct IO read patch, VFS layer calls
  * filesystem's direct_IO()
  * then grabs i_sem
  * followed by i_alloc_sem.
The above lock sequence would undoubtly cause deadlocks for external kernel
module that requires extra locking for various purposes. In GFS case, both
gfs_setattr() and gfs_direct_IO() need its own (global) locks to synchronize
inter-nodes (and inter-processes) control structures access. Two deadlocks
(bugzilla 171488, 173913 - bugzilla 173912 is for base kernel build) have been
found so far.

To work around this issue, a test kernel which forces GFS to take the two inode
semaphores in the earlier read code path was sent out before Thanksgiving break.
It reported back with successful results. The new RPMs require a base kernel
change (that allows GFS to bypass the inode semaphore acquiring within
__blockdev_direct_IO) - it basically does the following:

(vfs) sys_ftruncate()->do_truncate() {
                   -> call gfs_setattr -> g-lock ->g-unlock
(gfs) gfs_write (O_DIRECT) {
        __blockdev_direct_IO ->
                   new DIO_GFS_LOCKING {
                           don't get the inode semaphores }
(gfs) gfs_read (O_DIRECT) {
        _blockdev_direct_IO ->
                  new DIO_GFS_LOCKING {
                          don't get the inode sempahore }
appreciate if this patch could make into U3 - comment ? 

Comment 2 Wendy Cheng 2005-11-29 21:22:18 UTC
This bugzilla was opened with an expectation that we needed to change base
kernel to workaround the above deadlock. However, after further investigation,
we've found if GFS switches from DIO_LOCKING to DIO_NO_LOCKING (that was
currently used by raw device), we can workaround the deadlock.

So I'm closing this as an non-bug. The GFS changes will use bugzilla 173913. 

Comment 3 Wendy Cheng 2005-12-02 19:47:39 UTC
Unfortunately, we have to come back to this issue. We can't work around with
DIO_NO_LOCKING. The request has been sent to kernel group for review. Waiting
for the response. 

Comment 4 Wendy Cheng 2005-12-02 19:50:04 UTC
Created attachment 121777 [details]

New patch submitted to kernel group on 12/02/05.

Comment 9 Red Hat Bugzilla 2006-03-07 20:52:53 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.