Red Hat Bugzilla – Bug 147656
Known race in __wait_on_buffer()
Last modified: 2007-11-30 17:06:54 EST
Description of problem:
Customer is seeing a known race in __wait_on_buffer(). Applying the backported
patch solves the problem.
Effectively, more and more processes lock up in __wait_on_buffer() or
__lock_page(). A quick google finds http://lkml.org/lkml/2002/11/11/251 where
Andrea describes the bug and provides a patch. Manish and I backported it and
added some work for submit_bh_linked(). Patch is attached.
Note that this patch is in RHEL3. However, the parts for submit_bh_linked() are
Steps to Reproduce:
1. Run a big Oracle rman restore.
Dual HT 3GHz Xeon machine.
Created attachment 110910 [details]
Patch for the wait_on_buffer race.
Does this have a corresponding Issue Tracker ID number?
Issue Tracker 65672 created.
I built an AS2.1 kernel that includes the missed wakeup patch.
Please try it out and see if if fixes this problem:
Its located here: http://people.redhat.com/~lwoodman/.for_oracle/
Larry, when you say "missed wakeup patch," do you mean the patch I posted in
this bug, or do you mean something else?
I'm currently querying the customer to see if the environment is still available.
I also move the src.rpm to location:
The patch is number 99999 in the kernel spec file.
The customer ran the kernel, and found it to be unstable:
At about 21:00PM, all oracle process crashed. There
were a lot of UNDERRUN message in /var/log/messages.
Hopefully I can get the full /var/log/messages to attach here. The customer has
gone back to my kernel (e.57 + this patch), which has been working for them.
Created attachment 113442 [details]
messages file from the customer's first node. Contains the boots using the Oracle.62 kernel
This is the messages file from the customer's first node, tsintao. The log of
boots with the Oracle.62 kernel are here. The qla driver is reporting UNDERRUN
conditions. The customer says that Oracle crashed quickly with this kernel
after the UNDERRUN messages started appearing.
Created attachment 113443 [details]
/var/log/messages from the customer's other node, running the Oracle.62 kernel
This is the other node in the customer's Oracle RAC. Note the qla driver
messages appear here as well.
Joel, the official AS2.1-U7 kernel is located in:
It does not contain the wait_on_buffer wait patch but I need to know if this
kernel fixes the stability issues you experienced in the previous kernel that
I have similar problems on a DL580 G2 with oracle RAC and a VA7410. We are
running kernel 2.4.9-e.49enterprise.
This is the output of ps and sysrq tasks for one of the processes:
000 D oracle 22168 1 0 76 0 - 351004 wait_o Jan18 ? 00:04:57
oracle D 00000000 2400 22168 1 21284 22162 (NOTLB)
Call Trace: [<c012101b>] do_softirq [kernel] 0x7b (0xcdca1d48)
[<c0148376>] __wait_on_buffer [kernel] 0x76 (0xcdca1d68)
[<c0149429>] fsync_inode_buffers [kernel] 0x129 (0xcdca1dac)
[<f8837e8f>] journal_get_write_access_Rsmp_a7d05437 [jbd] 0x3f (0xcdca1dcc)
[<f88383f1>] journal_dirty_metadata_Rsmp_08cf9292 [jbd] 0x61 (0xcdca1dd8)
[<f884bb07>] ext3_do_update_inode [ext3] 0x287 (0xcdca1df0)
[<f884bb4a>] ext3_do_update_inode [ext3] 0x2ca (0xcdca1df4)
[<f8837e8f>] journal_get_write_access_Rsmp_a7d05437 [jbd] 0x3f (0xcdca1dfc)
[<f884beb5>] ext3_mark_iloc_dirty [ext3] 0x25 (0xcdca1e24)
[<f884bec6>] ext3_mark_iloc_dirty [ext3] 0x36 (0xcdca1e2c)
[<f883dcc7>] __jbd_kmalloc [jbd] 0x27 (0xcdca1e34)
[<f8838931>] journal_stop_Rsmp_5858b5e4 [jbd] 0x1a1 (0xcdca1e5c)
[<f884c069>] ext3_dirty_inode [ext3] 0xa9 (0xcdca1e7c)
[<c012d367>] vmtruncate [kernel] 0x1f7 (0xcdca1e90)
[<c015c7aa>] __mark_inode_dirty [kernel] 0x2a (0xcdca1ea0)
[<c015e85d>] inode_setattr [kernel] 0xcd (0xcdca1eb4)
[<f884bddf>] ext3_setattr [ext3] 0x22f (0xcdca1ed0)
[<c013a087>] deactivate_page [kernel] 0x17 (0xcdca1ef0)
[<c015e961>] notify_change [kernel] 0x81 (0xcdca1f04)
[<c015eaba>] notify_change [kernel] 0x1da (0xcdca1f08)
[<c014582d>] do_truncate [kernel] 0x6d (0xcdca1f34)
[<c015b3bc>] dput [kernel] 0x1c (0xcdca1f40)
[<f8847a14>] ext3_release_file [ext3] 0x14 (0xcdca1f58)
[<c01480e8>] __fput [kernel] 0x68 (0xcdca1f64)
[<f8847b0e>] ext3_sync_file [ext3] 0x4e (0xcdca1f88)
[<c014892d>] sys_fsync [kernel] 0x5d (0xcdca1f98)
[<c01073e3>] system_call [kernel] 0x33 (0xcdca1fc0)
Is there any kernel update solving the problem?
We can update to last kernel 2.4.9-e.65 if you need more tests.
Created attachment 124045 [details]
hang processes and sysrq output
This is the information about oracle hang processes after doing a rman restore.
This situation occurs every time we do a restore and the database is up in the