Bug 720115

Summary: I/O blocks other activity
Product: Red Hat Enterprise Linux 5 Reporter: sean fulton <sean>
Component: kernelAssignee: Red Hat Kernel Manager <kernel-mgr>
Status: CLOSED WONTFIX QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.6CC: sean
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-02 13:21:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description sean fulton 2011-07-09 15:50:25 UTC 
I have been using OpenVZ kernels so I thought it was an issue with OpenVZ, but after seeing several threads like this:

http://archimedeseureka.blogspot.com/2011/04/xen-guest-io-issue-task-blocked-for.html
http://bugs.centos.org/view.php?id=4515
https://bugzilla.redhat.com/show_bug.cgi?id=605444

I realize this is probably further upstream. I have been able to reproduce this issue for more than a year on a variety of hardware, both in single-drive and raid1 (software raid) platforms with SATA drives.  

What happens is during an intense I/O operation, server will lock up, blocking I/O to the disk. Server load will climb (I got over 100 on an 8-core machine) until everything stops. Killing the I/O process will slowly restore the machine to normal.

The most recent activity was duplicating a large database in one of my OpenVZ virtual servers like this:

mysqldump largedb | mysql newdb

Previously, I have done it by executing:

zcat largedb.sql.gz | mysql largedb

What is strange is that through the various OpenVZ kernel release cycles, the problems seems be in one kernel version, go away in a new release, then re-appear in a subsequent release.

Most recently I've found it in the following OpenVZ kernels:

vmlinuz-2.6.18-238.9.1.el5.028stab089.1
vmlinuz-2.6.18-238.12.1.el5.028stab091.1
vmlinuz-2.6.18-238.5.1.el5.028stab088.1

and two kernels based on the 194 branch of the RHEL5 kernel.

I just downgraded to
vmlinux-2.6.18-164.15.1.el5.028stab068.9

And everything is working fine, the system load is up to 4 on an 8-core machine while trying the operation above.

As I said, I think this may be a RedHat issue based on some of the bugs I've seen reported. Other folks reported that the issue was fixed by upgrading the firmware on their HP drive controllers, so the tickets were closed, but I'm not using HP hardware or any RAID HW controllers, and yet I am having similar issues, so I think it may be kernel related.

Here is the latest set of errors:
Jul  8 19:00:50 server-5 kernel: INFO: task kjournald:6192 blocked for more than 300 seconds.
Jul  8 19:00:50 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:00:50 server-5 kernel: kjournald     D ffff81041df10220     0  6192    147          6198  6158 (L-TLB)
Jul  8 19:00:50 server-5 kernel:  ffff810418971d10 0000000000000046 0000000300000000 ffff81041d9b2150
Jul  8 19:00:50 server-5 kernel:  ffff81041df10220 ffff81041e3780a0 0001a42c65b3cb40 00046076648a1da0
Jul  8 19:00:50 server-5 kernel:  ffff81041df10428 ffff81041e374000 ffffffff8031eb80 ffffffff8006fb96
Jul  8 19:00:50 server-5 kernel: Call Trace:
Jul  8 19:00:50 server-5 kernel:  [<ffffffff8006fb96>] do_gettimeofday+0x40/0x8f
Jul  8 19:00:50 server-5 kernel:  [<ffffffff80015372>] sync_buffer+0x0/0x40
Jul  8 19:00:50 server-5 kernel:  [<ffffffff80064c4b>] io_schedule+0x59/0x8a
Jul  8 19:00:50 server-5 kernel:  [<ffffffff800153ad>] sync_buffer+0x3b/0x40
Jul  8 19:00:50 server-5 kernel:  [<ffffffff80064e80>] __wait_on_bit+0x40/0x6e
Jul  8 19:00:50 server-5 kernel:  [<ffffffff80015372>] sync_buffer+0x0/0x40
Jul  8 19:00:55 server-5 kernel:  [<ffffffff80064f1a>] out_of_line_wait_on_bit+0x6c/0x78
Jul  8 19:00:58 server-5 kernel:  [<ffffffff800a314d>] wake_bit_function+0x0/0x23
Jul  8 19:00:59 server-5 kernel:  [<ffffffff88033711>] :jbd:journal_commit_transaction+0x66b/0x163e
Jul  8 19:01:01 server-5 kernel:  [<ffffffff8003f83d>] lock_timer_base+0x1b/0x3c
Jul  8 19:01:02 server-5 kernel:  [<ffffffff8004d6c5>] try_to_del_timer_sync+0x7f/0x88
Jul  8 19:01:06 server-5 kernel:  [<ffffffff800a2f07>] keventd_create_kthread+0x0/0xc4
Jul  8 19:01:07 server-5 kernel:  [<ffffffff88038269>] :jbd:kjournald+0xb9/0x21b
Jul  8 19:01:08 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:09 server-5 kernel:  [<ffffffff880381b0>] :jbd:kjournald+0x0/0x21b
Jul  8 19:01:11 server-5 kernel:  [<ffffffff8003399a>] kthread+0xfe/0x132
Jul  8 19:01:13 server-5 kernel:  [<ffffffff80061001>] child_rip+0xa/0x11
Jul  8 19:01:15 server-5 kernel:  [<ffffffff800a2f07>] keventd_create_kthread+0x0/0xc4
Jul  8 19:01:21 server-5 kernel:  [<ffffffff8003389c>] kthread+0x0/0x132
Jul  8 19:01:24 server-5 kernel:  [<ffffffff80060ff7>] child_rip+0x0/0x11
Jul  8 19:01:27 server-5 kernel: 
Jul  8 19:01:28 server-5 kernel: INFO: task syslogd:10648 blocked for more than 300 seconds.
Jul  8 19:01:30 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:31 server-5 kernel: syslogd       D ffff81041aa70220     0 10648   8793         10703       (NOTLB)
Jul  8 19:01:32 server-5 kernel:  ffff8103ff09dd68 0000000000000086 ffff8103ff09dd78 ffffffff800643c4
Jul  8 19:01:33 server-5 kernel:  ffff81041aa70220 ffff8103ce378620 0001a42b3bac4bf2 0004607349cb7388
Jul  8 19:01:35 server-5 kernel:  ffff81041aa70428 ffff8103dc3ee000 ffff81040c9492c0 ffff81041e3780a0
Jul  8 19:01:36 server-5 kernel: Call Trace:
Jul  8 19:01:37 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:39 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:40 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:42 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:43 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:45 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:46 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:48 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:49 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:50 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:53 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:54 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:54 server-5 kernel: 
Jul  8 19:01:54 server-5 kernel: INFO: task syslogd:18706 blocked for more than 300 seconds.
Jul  8 19:01:54 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:54 server-5 kernel: syslogd       D ffff8103fafb8460     0 18706  13072        630789       (NOTLB)
Jul  8 19:01:55 server-5 kernel:  ffff8103f1ed1d68 0000000000000086 ffff8103f1ed1d78 ffffffff800643c4
Jul  8 19:01:55 server-5 kernel:  ffff8103fafb8460 ffff8103d5ef01e0 0001a42b3bb3048a 0004607349dd6100
Jul  8 19:01:55 server-5 kernel:  ffff8103fafb8668 ffff8103f22c6000 ffff8103fe528b80 ffff8103e4a2a2a0
Jul  8 19:01:55 server-5 kernel: Call Trace:
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:55 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:55 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:55 server-5 kernel: 
Jul  8 19:01:55 server-5 kernel: INFO: task syslogd:20750 blocked for more than 300 seconds.
Jul  8 19:01:55 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:55 server-5 kernel: syslogd       D ffff8103f1a75560     0 20750  14580         20893       (NOTLB)
Jul  8 19:01:55 server-5 kernel:  ffff8103ed823d68 0000000000000086 ffff8103ed823d78 ffffffff800643c4
Jul  8 19:01:55 server-5 kernel:  ffff8103f1a75560 ffff8103bf7c0a60 0001a42b3bac47b7 0004607349cb66a8
Jul  8 19:01:55 server-5 kernel:  ffff8103f1a75768 ffff8103ac85e000 ffff8103fd67e180 ffff81041dc06120
Jul  8 19:01:55 server-5 kernel: Call Trace:
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:55 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:55 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:55 server-5 kernel: 
Jul  8 19:01:55 server-5 kernel: INFO: task syslogd:22940 blocked for more than 300 seconds.
Jul  8 19:01:55 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:55 server-5 kernel: syslogd       D ffff8103e4a2a2a0     0 22940  17310        631340       (NOTLB)
Jul  8 19:01:55 server-5 kernel:  ffff8103e4683d68 0000000000000082 ffff8103e4683d78 ffffffff800643c4
Jul  8 19:01:55 server-5 kernel:  ffff8103e4a2a2a0 ffff8103fafb8460 0001a42b3bb2f677 0004607349dd3b70
Jul  8 19:01:55 server-5 kernel:  ffff8103e4a2a4a8 ffff8103fd658000 ffff8103f3c11180 ffff8103f1c52ca0
Jul  8 19:01:55 server-5 kernel: Call Trace:
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:55 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:55 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:55 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 
Jul  8 19:01:56 server-5 kernel: INFO: task mysqld:428508 blocked for more than 300 seconds.
Jul  8 19:01:56 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:56 server-5 kernel: mysqld        D ffff8103c52574a0     0 428508  23107        672429 160247 (NOTLB)
Jul  8 19:01:56 server-5 kernel:  ffff81028f95dd68 0000000000000082 ffff81028f95dd78 ffffffff800643c4
Jul  8 19:01:56 server-5 kernel:  ffff8103c52574a0 ffff8103ffd62c20 0001a42b3bc8c63d 000460734a176590
Jul  8 19:01:56 server-5 kernel:  ffff8103c52576a8 ffff8103ac85e000 ffff8103fd67e180 ffff81041b8588c0
Jul  8 19:01:56 server-5 kernel: Call Trace:
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:56 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:56 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 
Jul  8 19:01:56 server-5 kernel: INFO: task syslogd:27026 blocked for more than 300 seconds.
Jul  8 19:01:56 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:56 server-5 kernel: syslogd       D ffff81041b9a0e20     0 27026  20785         27096 24775 (NOTLB)
Jul  8 19:01:56 server-5 kernel:  ffff8103d329bd68 0000000000000082 ffff8103d329bd78 ffffffff800643c4
Jul  8 19:01:56 server-5 kernel:  ffff81041b9a0e20 ffff8103c5266420 0001a42b3bac8e37 0004607349cc2448
Jul  8 19:01:56 server-5 kernel:  ffff81041b9a1028 ffff8103ed902000 ffff8103f24e6dc0 ffff8103ce378620
Jul  8 19:01:56 server-5 kernel: Call Trace:
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:56 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 
Jul  8 19:01:56 server-5 kernel: INFO: task mysqld:31321 blocked for more than 300 seconds.
Jul  8 19:01:56 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:56 server-5 kernel: mysqld        D ffff8103c5266420     0 31321  29267         31322 31320 (NOTLB)
Jul  8 19:01:56 server-5 kernel:  ffff8103b2f91d68 0000000000000082 ffff8103b2f91d78 ffffffff800643c4
Jul  8 19:01:56 server-5 kernel:  ffff8103c5266420 ffff8103c52574a0 0001a42b3bac9b1b 0004607349cc4688
Jul  8 19:01:56 server-5 kernel:  ffff8103c5266628 ffff8103fa992000 ffff8103f24e6dc0 ffff81041b9a0e20
Jul  8 19:01:56 server-5 kernel: Call Trace:
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:56 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 
Jul  8 19:01:56 server-5 kernel: INFO: task mysqld:31191 blocked for more than 300 seconds.
Jul  8 19:01:56 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:56 server-5 kernel: mysqld        D ffff8103ce378620     0 31191  30769         31203 31186 (NOTLB)
Jul  8 19:01:56 server-5 kernel:  ffff8103b669fd68 0000000000000086 ffff8103b669fd78 ffffffff800643c4
Jul  8 19:01:56 server-5 kernel:  ffff8103ce378620 ffff81041b9a0e20 0001a42b3bac7886 0004607349cbe8d8
Jul  8 19:01:56 server-5 kernel:  ffff8103ce378828 ffff8103ed902000 ffff8103ec069dc0 ffff81041aa70220
Jul  8 19:01:56 server-5 kernel: Call Trace:
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:56 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:56 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 
Jul  8 19:01:56 server-5 kernel: INFO: task sendmail:629665 blocked for more than 300 seconds.
Jul  8 19:01:56 server-5 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  8 19:01:56 server-5 kernel: sendmail      D ffff81015c42a5a0     0 629665  13072               630790 (NOTLB)
Jul  8 19:01:56 server-5 kernel:  ffff810206e35d68 0000000000000082 ffff810206e35d78 ffffffff800643c4
Jul  8 19:01:56 server-5 kernel:  ffff81015c42a5a0 ffff81041dc06120 0001a42b3bb53d7f 0004607349e34e30
Jul  8 19:01:56 server-5 kernel:  ffff81015c42a7a8 ffff81041dc02000 ffff8103fe528b80 ffff8100158e33e0
Jul  8 19:01:56 server-5 kernel: Call Trace:
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800643c4>] thread_return+0x6a/0x177
Jul  8 19:01:56 server-5 kernel:  [<ffffffff880377a5>] :jbd:log_wait_commit+0xa3/0xf5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800a311f>] autoremove_wake_function+0x0/0x2e
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80099185>] process_timeout+0x0/0x5
Jul  8 19:01:56 server-5 kernel:  [<ffffffff88030b6f>] :jbd:journal_stop+0x267/0x29b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80030fed>] __writeback_single_inode+0x1d9/0x318
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80103493>] sync_inode+0x24/0x34
Jul  8 19:01:56 server-5 kernel:  [<ffffffff8804d408>] :ext3:ext3_sync_file+0x118/0x144
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80064f6f>] mutex_lock+0xd/0x1d
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80052995>] do_fsync+0x9f/0x12b
Jul  8 19:01:56 server-5 kernel:  [<ffffffff800ef905>] __do_fsync+0x47/0x59
Jul  8 19:01:56 server-5 kernel:  [<ffffffff80062766>] ia32_sysret+0x0/0x5
Jul  8 19:01:56 server-5 kernel: 

Any help would be appreciated.

sean
Comment 1 RHEL Program Management 2014-03-07 13:47:10 UTC 
This bug/component is not included in scope for RHEL-5.11.0 which is the last RHEL5 minor release. This Bugzilla will soon be CLOSED as WONTFIX (at the end of RHEL5.11 development phase (Apr 22, 2014)). Please contact your account manager or support representative in case you need to escalate this bug.
Comment 2 RHEL Program Management 2014-06-02 13:21:29 UTC 
Thank you for submitting this request for inclusion in Red Hat Enterprise Linux 5. We've carefully evaluated the request, but are unable to include it in RHEL5 stream. If the issue is critical for your business, please provide additional business justification through the appropriate support channels (https://access.redhat.com/site/support).