Bug 147656 - Known race in __wait_on_buffer()
Known race in __wait_on_buffer()
Status: CLOSED DEFERRED
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: kernel (Show other bugs)
2.1
All Linux
medium Severity high
: ---
: ---
Assigned To: Larry Woodman
Brian Brock
:
Depends On:
Blocks: 143573
  Show dependency treegraph
 
Reported: 2005-02-09 22:08 EST by Joel Becker
Modified: 2007-11-30 17:06 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-09-13 16:36:19 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch for the wait_on_buffer race. (5.49 KB, patch)
2005-02-09 22:08 EST, Joel Becker
no flags Details | Diff
messages file from the customer's first node. Contains the boots using the Oracle.62 kernel (640.60 KB, text/plain)
2005-04-20 19:18 EDT, Joel Becker
no flags Details
/var/log/messages from the customer's other node, running the Oracle.62 kernel (788.66 KB, text/plain)
2005-04-20 19:21 EDT, Joel Becker
no flags Details
hang processes and sysrq output (96.74 KB, text/plain)
2006-02-02 07:27 EST, Enrique Galdu
no flags Details

  None (edit)
Description Joel Becker 2005-02-09 22:08:28 EST
Description of problem:
Customer is seeing a known race in __wait_on_buffer().  Applying the backported
patch solves the problem.

Effectively, more and more processes lock up in __wait_on_buffer() or
__lock_page().  A quick google finds http://lkml.org/lkml/2002/11/11/251 where
Andrea describes the bug and provides a patch.  Manish and I backported it and
added some work for submit_bh_linked().  Patch is attached.

Note that this patch is in RHEL3.  However, the parts for submit_bh_linked() are
not.

How reproducible:
Always reproducable

Steps to Reproduce:
1. Run a big Oracle rman restore.
  
Actual results:
Hang.

Expected results:
No Hang.

Additional info:
Dual HT 3GHz Xeon machine.
8GB RAM
Comment 1 Joel Becker 2005-02-09 22:08:28 EST
Created attachment 110910 [details]
Patch for the wait_on_buffer race.
Comment 2 Suzanne Hillman 2005-02-10 14:48:05 EST
Does this have a corresponding Issue Tracker ID number?
Comment 3 Van Okamura 2005-02-10 20:59:38 EST
Issue Tracker 65672 created.
Comment 5 Larry Woodman 2005-04-12 13:37:03 EDT
I built an AS2.1 kernel that includes the missed wakeup patch.
Please try it out and see if if fixes this problem:

Its located here: http://people.redhat.com/~lwoodman/.for_oracle/


Thanks, Larry


Comment 6 Joel Becker 2005-04-12 16:47:09 EDT
Larry, when you say "missed wakeup patch," do you mean the patch I posted in
this bug, or do you mean something else?

I'm currently querying the customer to see if the environment is still available.
Comment 7 Larry Woodman 2005-04-13 10:41:35 EDT
Yes Joel.

Larry
Comment 8 Larry Woodman 2005-04-13 15:55:31 EDT
I also move the src.rpm to location:

>>>http://people.redhat.com/~lwoodman/.for_oracle/

The patch is number 99999 in the kernel spec file.


Larry
Comment 9 Joel Becker 2005-04-18 18:13:53 EDT
The customer ran the kernel, and found it to be unstable:

    At about 21:00PM, all oracle process crashed.  There
    were a lot of UNDERRUN message in /var/log/messages.

Hopefully I can get the full /var/log/messages to attach here.  The customer has
gone back to my kernel (e.57 + this patch), which has been working for them.
Comment 10 Joel Becker 2005-04-20 19:18:36 EDT
Created attachment 113442 [details]
messages file from the customer's first node.  Contains the boots using the Oracle.62 kernel

This is the messages file from the customer's first node, tsintao.  The log of
boots with the Oracle.62 kernel are here.  The qla driver is reporting UNDERRUN
conditions.  The customer says that Oracle crashed quickly with this kernel
after the UNDERRUN messages started appearing.
Comment 11 Joel Becker 2005-04-20 19:21:36 EDT
Created attachment 113443 [details]
/var/log/messages from the customer's other node, running the Oracle.62 kernel

This is the other node in the customer's Oracle RAC.  Note the qla driver
messages appear here as well.
Comment 13 Larry Woodman 2005-05-04 13:53:38 EDT
Joel, the official AS2.1-U7 kernel is located in:

>>>http://people.redhat.com/~lwoodman/AS2.1/

It does not contain the wait_on_buffer wait patch but I need to know if this
kernel fixes the stability issues you experienced in the previous kernel that
you tried.

Larry
Comment 15 Enrique Galdu 2006-01-26 06:49:55 EST
I have similar problems on a DL580 G2 with oracle RAC and a VA7410. We are
running kernel 2.4.9-e.49enterprise.

This is the output of ps and sysrq tasks for one of the processes:
ps
000 D oracle   22168     1  0  76   0    - 351004 wait_o Jan18 ?       00:04:57
oracledss1 (LOCAL=NO)

sysrq tasks
oracle        D 00000000  2400 22168      1         21284 22162 (NOTLB)
Call Trace: [<c012101b>] do_softirq [kernel] 0x7b (0xcdca1d48)
[<c0148376>] __wait_on_buffer [kernel] 0x76 (0xcdca1d68)
[<c0149429>] fsync_inode_buffers [kernel] 0x129 (0xcdca1dac)
[<f8837e8f>] journal_get_write_access_Rsmp_a7d05437 [jbd] 0x3f (0xcdca1dcc)
[<f88383f1>] journal_dirty_metadata_Rsmp_08cf9292 [jbd] 0x61 (0xcdca1dd8)
[<f884bb07>] ext3_do_update_inode [ext3] 0x287 (0xcdca1df0)
[<f884bb4a>] ext3_do_update_inode [ext3] 0x2ca (0xcdca1df4)
[<f8837e8f>] journal_get_write_access_Rsmp_a7d05437 [jbd] 0x3f (0xcdca1dfc)
[<f884beb5>] ext3_mark_iloc_dirty [ext3] 0x25 (0xcdca1e24)
[<f884bec6>] ext3_mark_iloc_dirty [ext3] 0x36 (0xcdca1e2c)
[<f883dcc7>] __jbd_kmalloc [jbd] 0x27 (0xcdca1e34)
[<f8838931>] journal_stop_Rsmp_5858b5e4 [jbd] 0x1a1 (0xcdca1e5c)
[<f884c069>] ext3_dirty_inode [ext3] 0xa9 (0xcdca1e7c)
[<c012d367>] vmtruncate [kernel] 0x1f7 (0xcdca1e90)
[<c015c7aa>] __mark_inode_dirty [kernel] 0x2a (0xcdca1ea0)
[<c015e85d>] inode_setattr [kernel] 0xcd (0xcdca1eb4)
[<f884bddf>] ext3_setattr [ext3] 0x22f (0xcdca1ed0)
[<c013a087>] deactivate_page [kernel] 0x17 (0xcdca1ef0)
[<c015e961>] notify_change [kernel] 0x81 (0xcdca1f04)
[<c015eaba>] notify_change [kernel] 0x1da (0xcdca1f08)
[<c014582d>] do_truncate [kernel] 0x6d (0xcdca1f34)
[<c015b3bc>] dput [kernel] 0x1c (0xcdca1f40)
[<f8847a14>] ext3_release_file [ext3] 0x14 (0xcdca1f58)
[<c01480e8>] __fput [kernel] 0x68 (0xcdca1f64)
[<f8847b0e>] ext3_sync_file [ext3] 0x4e (0xcdca1f88)
[<c014892d>] sys_fsync [kernel] 0x5d (0xcdca1f98)
[<c01073e3>] system_call [kernel] 0x33 (0xcdca1fc0)

Is there any kernel update solving the problem?
We can update to last kernel 2.4.9-e.65 if you need more tests.
Thanks.
Comment 16 Enrique Galdu 2006-02-02 07:27:03 EST
Created attachment 124045 [details]
hang processes and sysrq output

This is the information about oracle hang processes after doing a rman restore.
This situation occurs every time we do a restore and the database is up in the
same node.

Note You need to log in before you can comment on or make changes to this bug.