Bug 679790

Summary: intel-sunriseridge-01.lab.bos.redhat.com very low performance
Product: Red Hat Enterprise Linux 6 Reporter: Stefan Assmann <sassmann>
Component: kernelAssignee: Red Hat Kernel Manager <kernel-mgr>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.0CC: arozansk, pbunyan, prarit, rwheeler
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-26 17:00:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Stefan Assmann 2011-02-23 14:28:40 UTC 
Description of problem:
The machine feels slow at install time already as it takes very long for the standard installation to finish. Also general system performances is lacking. A kernel build with make -j32 often stalls completely!
Also while doing the kernel build I observed:

INFO: task jbd2/dm-0-8:1899 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
jbd2/dm-0-8   D ffff88027fe75000     0  1899      2 0x00000000
 ffff880273653d10 0000000000000046 ffff880273653cd8 ffff880273653cd4
 ffff8801e243ac90 ffff88027fe75000 ffff880287416980 0000000100204367
 ffff8802736a5ad8 ffff880273653fd8 0000000000010518 ffff8802736a5ad8
Call Trace:
 [<ffffffffa00618f0>] jbd2_journal_commit_transaction+0x1c0/0x14e0 [jbd2]
 [<ffffffff8107d5ac>] ? lock_timer_base+0x3c/0x70
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa00680b8>] kjournald2+0xb8/0x220 [jbd2]
 [<ffffffff81091ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa0068000>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81091936>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20
INFO: task flush-253:0:4600 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-253:0   D ffff88027fe74c00     0  4600      2 0x00000080
 ffff880270dc7420 0000000000000046 0000000000000000 ffffffffa000471c
 ffff880270dc73a0 ffffffff814c9cf6 ffffffff8175fe80 0000000100204088
 ffff880270da0678 ffff880270dc7fd8 0000000000010518 ffff880270da0678
Call Trace:
 [<ffffffffa000471c>] ? dm_table_unplug_all+0x5c/0xd0 [dm_mod]
 [<ffffffff814c9cf6>] ? down_write+0x16/0x40
 [<ffffffff8119dcf0>] ? sync_buffer+0x0/0x50
 [<ffffffff814c8a23>] io_schedule+0x73/0xc0
 [<ffffffff8119dd30>] sync_buffer+0x40/0x50
 [<ffffffff814c929f>] __wait_on_bit+0x5f/0x90
 [<ffffffff8119dcf0>] ? sync_buffer+0x0/0x50
 [<ffffffff814c9348>] out_of_line_wait_on_bit+0x78/0x90
 [<ffffffff81091ce0>] ? wake_bit_function+0x0/0x50
 [<ffffffff8119dce6>] __wait_on_buffer+0x26/0x30
 [<ffffffffa00b927d>] ext4_mb_init_cache+0x34d/0x970 [ext4]
 [<ffffffff8110c3f9>] ? add_to_page_cache_locked+0xc9/0x140
 [<ffffffffa00b9974>] ext4_mb_init_group+0xd4/0x260 [ext4]
 [<ffffffffa00b9e8d>] ext4_mb_load_buddy+0x38d/0x3d0 [ext4]
 [<ffffffffa00bcc86>] ext4_mb_regular_allocator+0x1e6/0x550 [ext4]
 [<ffffffffa00037c7>] ? dm_request+0x1e7/0x230 [dm_mod]
 [<ffffffffa00bd37d>] ext4_mb_new_blocks+0x38d/0x560 [ext4]
 [<ffffffffa00b12be>] ? ext4_ext_find_extent+0x2be/0x320 [ext4]
 [<ffffffffa00b4230>] ext4_ext_get_blocks+0xfa0/0x1800 [ext4]
 [<ffffffff8124117f>] ? submit_bio+0x8f/0x120
 [<ffffffffa0092610>] ext4_get_blocks+0x1f0/0x2a0 [ext4]
 [<ffffffffa00934ec>] mpage_da_map_blocks+0xac/0x430 [ext4]
 [<ffffffffa009402e>] ext4_da_writepages+0x2be/0x600 [ext4]
 [<ffffffff81120951>] do_writepages+0x21/0x40
 [<ffffffff81195cdd>] writeback_single_inode+0xdd/0x2c0
 [<ffffffff811960de>] writeback_sb_inodes+0xce/0x180
 [<ffffffff81196233>] writeback_inodes_wb+0xa3/0x1a0
 [<ffffffff811965cb>] wb_writeback+0x29b/0x3f0
 [<ffffffff814c8286>] ? thread_return+0x4e/0x778
 [<ffffffff811968b9>] wb_do_writeback+0x199/0x240
 [<ffffffff811969c3>] bdi_writeback_task+0x63/0x1b0
 [<ffffffff81091b67>] ? bit_waitqueue+0x17/0xd0
 [<ffffffff8112ecd0>] ? bdi_start_fn+0x0/0x100
 [<ffffffff8112ed56>] bdi_start_fn+0x86/0x100
 [<ffffffff8112ecd0>] ? bdi_start_fn+0x0/0x100
 [<ffffffff81091936>] kthread+0x96/0xa0
 [<ffffffff810141ca>] child_rip+0xa/0x20
 [<ffffffff810918a0>] ? kthread+0x0/0xa0
 [<ffffffff810141c0>] ? child_rip+0x0/0x20

System was unreachable after that.

Version-Release number of selected component (if applicable):
2.6.32-71.el6.x86_64

How reproducible:
always 

Steps to Reproduce:
1. build kernel with make -j32
2.
3.
Comment 2 Prarit Bhargava 2011-02-25 17:41:12 UTC 
I installed RHEL6.0 (GA) on the system, checked out a kernel and watched a -j32 compile.  During the compile there are several "long hangs".

I cleaned the tree, and compiled again and did a 

echo 30 > /proc/sys/kernel/hung_task_timeout_secs

After doing that I see one or two messages similar to the ones above which seem to imply an issue with ext4 and the journaling code (AFAICT).

esandeen -- this *could* be a weird storage issue, but for now I'm assigning it to you.

Please note that this system is supposed to be converted into a Westmere system in the next few weeks, and we have NOT seen this issue on other Nehalem based boxes.

P.
Comment 3 Ric Wheeler 2011-03-17 19:21:20 UTC 
I think that this is more of a storage level issue and ext4 is just the thing waiting on something else.

It would be good to retry this on the current crop of 6.1 kernels....
Comment 4 RHEL Program Management 2011-10-07 15:23:35 UTC 
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.