Bug 650813
| Summary: | [ext4/xfstests] 232 made the system hung_task_timeout | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Igor Zhang <yugzhang> |
| Component: | kernel | Assignee: | Eric Sandeen <esandeen> |
| Status: | CLOSED WONTFIX | QA Contact: | Filesystem QE <fs-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 5.6 | CC: | branto, eguan, esandeen, kzhang, lczerner, lmcilroy, rwheeler, yugzhang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-06-02 13:04:22 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 657336 has been marked as a duplicate of this bug. *** This also happens with ext3, should I open a new bug for ext3? INFO: task kjournald:24338 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kjournald D ffffffff80154228 0 24338 211 24358 24163 (L-TLB) ffff810422c21dd0 0000000000000046 0000000000000100 0000000000000000 0000000000000000 000000000000000a ffff810203f5b7a0 ffff8102873217a0 000053aebd85f5a5 0000000000000952 ffff810203f5b988 0000000700000000 Call Trace: [<ffffffff88033661>] :jbd:journal_commit_transaction+0x173/0x10aa [<ffffffff800a2a26>] autoremove_wake_function+0x0/0x2e [<ffffffff8004ad9d>] try_to_del_timer_sync+0x7f/0x88 [<ffffffff88037662>] :jbd:kjournald+0xc1/0x213 [<ffffffff800a2a26>] autoremove_wake_function+0x0/0x2e [<ffffffff880375a1>] :jbd:kjournald+0x0/0x213 [<ffffffff800a280e>] keventd_create_kthread+0x0/0xc4 [<ffffffff80032734>] kthread+0xfe/0x132 [<ffffffff8005dfb1>] child_rip+0xa/0x11 [<ffffffff800a280e>] keventd_create_kthread+0x0/0xc4 [<ffffffff80032636>] kthread+0x0/0x132 [<ffffffff8005dfa7>] child_rip+0x0/0x11 INFO: task fsstress:24350 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. fsstress D ffffffff80154228 0 24350 24346 24352 (NOTLB) ffff810214a49c78 0000000000000086 ffff810214a49d78 ffffffff80106f9c 0000000010000042 0000000000000007 ffff81010e8a7100 ffff810287158860 000053aebd860ab7 000000000000271c ffff81010e8a72e8 0000000217be8a38 Call Trace: [<ffffffff80106f9c>] remove_tree+0x458/0x469 [<ffffffff88032008>] :jbd:start_this_handle+0x2e9/0x370 [<ffffffff800a2a26>] autoremove_wake_function+0x0/0x2e [<ffffffff8803215a>] :jbd:journal_start+0xcb/0x102 [<ffffffff88058202>] :ext3:ext3_release_dquot+0x44/0x82 [<ffffffff801042a4>] dqput+0x1c0/0x207 [<ffffffff80106299>] dquot_transfer+0x352/0x389 [<ffffffff8002cb11>] notify_change+0x1fd/0x2f5 [<ffffffff8003b2ae>] chown_common+0x9b/0xb0 [<ffffffff800e2211>] sys_lchown+0x38/0x53 [<ffffffff8005d229>] tracesys+0x71/0xe0 [<ffffffff8005d28d>] tracesys+0xd5/0xe0 This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.7 and Red Hat does not plan to fix this issue the currently developed update. Contact your manager or support representative in case you need to escalate this bug. I think there are known problems with extN & quotas in rhel5. There are upstream fixes which are quite invasive, and to my knowledge we've not had customer reports of this problem. So I'm on the fence about how aggressively to go after this one... -Eric This request was evaluated by Red Hat Product Management for inclusion in Red Hat Enterprise Linux 5.8 and Red Hat does not plan to fix this issue the currently developed update. Contact your manager or support representative in case you need to escalate this bug. This request was not resolved in time for the current release. Red Hat invites you to ask your support representative to propose this request, if still desired, for consideration in the next release of Red Hat Enterprise Linux. This bug/component is not included in scope for RHEL-5.11.0 which is the last RHEL5 minor release. This Bugzilla will soon be CLOSED as WONTFIX (at the end of RHEL5.11 development phase (Apr 22, 2014)). Please contact your account manager or support representative in case you need to escalate this bug. Thank you for submitting this request for inclusion in Red Hat Enterprise Linux 5. We've carefully evaluated the request, but are unable to include it in RHEL5 stream. If the issue is critical for your business, please provide additional business justification through the appropriate support channels (https://access.redhat.com/site/support). The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: [ext4/xfstests] 232 made the system hung_task_timeout # ./check 232 FSTYP -- ext4 PLATFORM -- Linux/i686 dell-pe6950-01 2.6.18-229.el5PAE MKFS_OPTIONS -- /dev/loop1 MOUNT_OPTIONS -- -o acl,user_xattr -o context=system_u:object_r:nfs_t:s0 /dev/loop1 /mnt/testarea/scratch 232 After a while on console, following messages emerged, INFO: task pdflush:10813 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. pdflush D 0000083F 2188 10813 35 14838 10808 (L-TLB) d2ef8bb0 00000046 1e85e00e 0000083f d1dbcaf4 df626d4c e30a5e3c 0000000a f7f27aa0 1e8ff1d0 0000083f 000a11c2 00000001 f7f27bac c5419ac4 f71f73c0 f731e000 e30a5e3c 00000000 00000030 00189adb d1dbca64 00000000 d2ef8d24 Call Trace: [<f8e83f07>] ext4_ext_insert_extent+0xb6f/0xbe3 [ext4] [<c061f981>] rwsem_down_read_failed+0x128/0x143 [<c04399b7>] .text.lock.rwsem+0x35/0x3a [<c04a2377>] dquot_claim_space+0x26/0x103 [<f8e71bed>] ext4_da_update_reserve_space+0x126/0x1ba [ext4] [<f8e8642a>] ext4_ext_get_blocks+0x14ef/0x1649 [ext4] [<c041fc31>] try_to_wake_up+0x3e8/0x3f2 [<c0477c75>] __find_get_block+0x15c/0x166 [<c0477caf>] __getblk+0x30/0x27a [<c041efe5>] complete+0x2b/0x3d [<f8a36425>] loop_make_request+0x95/0xc4 [loop] [<f8e8661b>] __ext4_handle_dirty_metadata+0x97/0xa8 [ext4] [<f8e71d7a>] ext4_get_blocks+0xf9/0x1a5 [ext4] [<f8e71f10>] mpage_da_map_blocks+0x9c/0x6dc [ext4] [<c045dc7f>] pagevec_lookup_tag+0x1b/0x22 [<c0496468>] write_cache_pages+0x132/0x2da [<f8e7570b>] ext4_da_writepages+0x336/0x4d4 [ext4] [<c045cc05>] do_writepages+0x20/0x32 [<c04948e7>] __writeback_single_inode+0x164/0x2a3 [<c045043a>] delayacct_end+0x58/0x7a [<c0494d08>] sync_sb_inodes+0x17c/0x21f [<c0494f57>] writeback_inodes+0x6a/0xb0 [<c045cf74>] background_writeout+0x71/0xc3 [<c045d4b5>] pdflush+0x0/0x1a3 [<c045d5c0>] pdflush+0x10b/0x1a3 [<c045cf03>] background_writeout+0x0/0xc3 [<c0436ebf>] kthread+0xc0/0xed [<c0436dff>] kthread+0x0/0xed [<c0405c87>] kernel_thread_helper+0x7/0x10 ======================= INFO: task jbd2/loop1-8:15001 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/loop1-8 D 0000083F 3088 15001 35 15002 14846 (L-TLB) e174eeec 00000046 1e93c35a 0000083f f6df7c54 c55d044c c544305c 0000000a f6eda000 1e93cd4b 0000083f 000009f1 00000000 f6eda10c c5412c80 f6d05200 c54137e0 00000000 00000000 c55d0400 00000000 00000000 e174ef38 ffffffff Call Trace: [<f8e14afe>] jbd2_journal_commit_transaction+0x14d/0xeff [jbd2] [<c061e4fa>] schedule+0x9c6/0xa4f [<c0436f83>] autoremove_wake_function+0x0/0x2d [<c042e2ef>] lock_timer_base+0x15/0x2f [<c042e36e>] try_to_del_timer_sync+0x65/0x6c [<f8e18802>] kjournald2+0x83/0x1a4 [jbd2] [<c0436f83>] autoremove_wake_function+0x0/0x2d [<f8e1877f>] kjournald2+0x0/0x1a4 [jbd2] [<c0436ebf>] kthread+0xc0/0xed [<c0436dff>] kthread+0x0/0xed [<c0405c87>] kernel_thread_helper+0x7/0x10 ======================= INFO: task fsstress:15025 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. fsstress D 0000083F 1936 15025 15020 15026 (NOTLB) ccf20ad0 00000086 1e86399c 0000083f 00010001 cf090400 f731e000 00000007 f6f03aa0 1e8f3e18 0000083f 0009047c 00000007 f6f03bac c544305c f6fac3c0 00000001 00000000 00000246 f731e000 cf7ba968 ccf20b48 f8e8da7f ffffffff Call Trace: [<f8e8da7f>] ext4_mb_new_blocks+0x3db/0x567 [ext4] [<c061f981>] rwsem_down_read_failed+0x128/0x143 [<c04399b7>] .text.lock.rwsem+0x35/0x3a [<c04a1de0>] __dquot_alloc_space+0x55/0x195 [<f8e6cccf>] ext4_new_meta_blocks+0xda/0x129 [ext4] [<f8e82bbd>] ext4_ext_new_meta_block+0x26/0x2c [ext4] [<f8e83be1>] ext4_ext_insert_extent+0x849/0xbe3 [ext4] [<f8e8634f>] ext4_ext_get_blocks+0x1414/0x1649 [ext4] [<c0477caf>] __getblk+0x30/0x27a [<c0457987>] find_get_page+0x18/0x3f [<c0477931>] __find_get_block_slow+0xfb/0x105 [<c0477c75>] __find_get_block+0x15c/0x166 [<f8e71d7a>] ext4_get_blocks+0xf9/0x1a5 [ext4] [<f8e71f10>] mpage_da_map_blocks+0x9c/0x6dc [ext4] [<c045dc7f>] pagevec_lookup_tag+0x1b/0x22 [<c0496468>] write_cache_pages+0x132/0x2da [<c045792a>] find_get_pages_tag+0x30/0x75 [<f8e7570b>] ext4_da_writepages+0x336/0x4d4 [ext4] [<c045cc05>] do_writepages+0x20/0x32 [<c04948e7>] __writeback_single_inode+0x164/0x2a3 [<c0458659>] wait_on_page_writeback_range+0xa6/0xf1 [<c0494d08>] sync_sb_inodes+0x17c/0x21f [<c0494e2a>] sync_inodes_sb+0x7f/0x8e [<c0494e81>] __sync_inodes+0x48/0x8b [<c0494ed5>] sync_inodes+0x11/0x29 [<c04782ae>] do_sync+0x11/0x55 [<c04782fc>] sys_sync+0xa/0xd [<c0404f4b>] syscall_call+0x7/0xb ======================= INFO: task fsstress:15026 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. fsstress D 0000083F 1936 15026 15020 15027 15025 (NOTLB) ccf1fee4 00000082 1e4f1036 0000083f 00000000 ccf1ff48 c045dc7f 00000007 f6c55000 1e4f444c 0000083f 00003416 00000001 f6c5510c c5419ac4 c554c740 00000000 c571c4f4 00000000 c047cab2 ffffffff 00000001 00000001 ccf1ff20 Call Trace: [<c045dc7f>] pagevec_lookup_tag+0x1b/0x22 [<c047cab2>] blkdev_writepage+0x0/0xc [<c048c618>] inode_wait+0x5/0x8 [<c061ed3d>] __wait_on_bit+0x33/0x58 [<c048c613>] inode_wait+0x0/0x8 [<c049487f>] __writeback_single_inode+0xfc/0x2a3 [<c045dc7f>] pagevec_lookup_tag+0x1b/0x22 [<c0436fb0>] wake_bit_function+0x0/0x3c [<c0494d08>] sync_sb_inodes+0x17c/0x21f [<c0494e2a>] sync_inodes_sb+0x7f/0x8e [<c0494e81>] __sync_inodes+0x48/0x8b [<c04782d2>] do_sync+0x35/0x55 [<c04782fc>] sys_sync+0xa/0xd [<c0404f4b>] syscall_call+0x7/0xb ======================= INFO: task fsstress:15027 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. fsstress D 0000083F 1936 15027 15020 15028 15026 (NOTLB) c7ce2bec 00000082 1e8f8d02 0000083f f6fac3c0 00000042 f6829000 00000007 f6829000 1e8f9b00 0000083f 00000dfe 00000007 f682910c c544305c f73c2c80 1e8721dd 00000000 0000de4c 00000004 f6829110 c542e590 f73c2c80 ffffffff Call Trace: [<c061f981>] rwsem_down_read_failed+0x128/0x143 [<c04399b7>] .text.lock.rwsem+0x35/0x3a [<c04a1de0>] __dquot_alloc_space+0x55/0x195 [<f8e74282>] ext4_da_get_block_prep+0x23a/0x373 [ext4] [<c0478a1d>] __block_prepare_write+0x19c/0x381 [<c045bf2e>] get_page_from_freelist+0x96/0x370 [<c047819d>] block_write_end+0x48/0x50 [<c0478ca5>] block_write_begin+0x80/0xdf [<f8e74048>] ext4_da_get_block_prep+0x0/0x373 [ext4] [<f8e75a93>] ext4_da_write_begin+0x1ea/0x28b [ext4] [<f8e74048>] ext4_da_get_block_prep+0x0/0x373 [ext4] [<c0458ece>] generic_file_buffered_write+0x101/0x58b [<f8e13499>] jbd2_journal_stop+0x175/0x17f [jbd2] [<c04597fe>] __generic_file_aio_write_nolock+0x4a6/0x52a [<c04c822b>] avc_has_perm+0x3c/0x46 [<c04598d9>] generic_file_aio_write+0x57/0xaa [<f8e6dbd6>] ext4_file_write+0x89/0x112 [ext4] [<c047593a>] do_sync_write+0xb6/0xf1 [<c0436f83>] autoremove_wake_function+0x0/0x2d [<c0475884>] do_sync_write+0x0/0xf1 [<c04761c3>] vfs_write+0xa1/0x143 [<c04767ed>] sys_write+0x3c/0x63 [<c0404f4b>] syscall_call+0x7/0xb ======================= ... Version-Release number of selected component (if applicable): kernel 2.6.18-230.el5 on different architectures How reproducible: Often Steps to Reproduce: 1../check 232 2. 3. Actual results: 232 blocked itself and other system processes. Expected results: No processes blocked. Additional info: Related failed beaker jobs, i386 https://beaker.engineering.redhat.com/recipes/57215 https://beaker.engineering.redhat.com/logs/2010/11/287/28778/57215/647148/2002659///console_dmesg--kernel-filesystems-xfs-xfstests-LOCALWATCHDOG.log x86_64 https://beaker.engineering.redhat.com/recipes/57222 https://beaker.engineering.redhat.com/logs/2010/11/287/28778/57222/647169/2004520///console_dmesg--kernel-filesystems-xfs-xfstests-LOCALWATCHDOG.log ia64 https://beaker.engineering.redhat.com/recipes/57230 https://beaker.engineering.redhat.com/logs/2010/11/287/28778/57230/647194/2004260///console_dmesg--kernel-filesystems-xfs-xfstests-LOCALWATCHDOG.log ppc64 https://beaker.engineering.redhat.com/recipes/57243 https://beaker.engineering.redhat.com/logs/2010/11/287/28778/57243/647235/2002733///console_dmesg--kernel-filesystems-xfs-xfstests-LOCALWATCHDOG.log