Bug 1356396
| Summary: | [crash kernel panic] Internal error XFS_WANT_CORRUPTED_RETURN | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Qiao Zhao <qzhao> | |
| Component: | kexec-tools | Assignee: | kdump team <kdump-team-bugs> | |
| Status: | CLOSED DUPLICATE | QA Contact: | Qiao Zhao <qzhao> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 7.3 | CC: | chorn, dchinner, esandeen, lilu, myamazak, panand, qzhao, ruyang, tonishim, xlpang | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1383573 (view as bug list) | Environment: | ||
| Last Closed: | 2016-10-11 08:26:13 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1383573 | |||
Full log in here: http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2016/07/14016/1401642/2866536/console.log Interesting, it is the first time we met a corruption. Dave, any thoughts? NOthing to go on here. There's a corruption detected during log recovery, which means something has gone wrong before this filesystem was mounted. Keep in mind that "kdump testing" means nothing to me - I've got no idea what testing is being performed what platform, the storag ein use, the fielsystem configuration, the context, what filesystem had the error, what happened prior to triggering kdump etc. There's not much we can do with just a random "something got corrupted" reports like this. -Dave. Qiao, could you check if you can reproduce it reliability? If so please provide enough detail information as Dave replied in comment #3. (In reply to Dave Young from comment #4) > Qiao, could you check if you can reproduce it reliability? If so please > provide enough detail information as Dave replied in comment #3. Hi Dave, I met this problem again, please look: job: https://beaker.engineering.redhat.com/recipes/2872991 console log: http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2016/07/14043/1404381/2872991/console.log ] Started File System Check on /dev/d...ed0-a3ac-4ee7-855d-c84c86cef3a3. Mounting /sysroot... [ 6.184053] SGI XFS with ACLs, security attributes, no debug enabled [ 6.194266] XFS (sda9): Mounting V5 Filesystem [ 6.324227] XFS (sda9): Starting recovery (logdev: internal) [ 6.441954] XFS (sda9): Internal error XFS_WANT_CORRUPTED_RETURN at line 188 of file fs/xfs/libxfs/xfs_dir2_data.c. Caller xfs_dir3_block_verify+0x9a/0xb0 [xfs] [ 6.458086] CPU: 11 PID: 516 Comm: mount Not tainted 3.10.0-469.el7.x86_64 #1 [ 6.466059] Hardware name: IBM System x3650 M4 -[7915IK1]-/00W2609, BIOS -[VVE120EUS-1.21]- 10/11/2012 [ 6.476457] ffff88086b520cd0 00000000b10406e0 ffff88086c0476f8 ffffffff81684ee9 [ 6.484767] ffff88086c047710 ffffffffa032391b ffffffffa030399a ffff88086c047798 [ 6.493075] ffffffffa0305c69 ffff88086b520ff8 ffff88086d7cd580 00000009812e995e [ 6.501381] Call Trace: [ 6.504122] [<ffffffff81684ee9>] dump_stack+0x19/0x1b [ 6.509885] [<ffffffffa032391b>] xfs_error_report+0x3b/0x40 [xfs] [ 6.516803] [<ffffffffa030399a>] ? xfs_dir3_block_verify+0x9a/0xb0 [xfs] [ 6.524399] [<ffffffffa0305c69>] __xfs_dir3_data_check+0x459/0x560 [xfs] [ 6.531995] [<ffffffffa030399a>] xfs_dir3_block_verify+0x9a/0xb0 [xfs] [ 6.539394] [<ffffffffa0303ba4>] xfs_dir3_block_write_verify+0x44/0x110 [xfs] [ 6.547466] [<ffffffffa0321418>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] [ 6.555449] [<ffffffffa031f7e1>] _xfs_buf_ioapply+0x91/0x460 [xfs] [ 6.562443] [<ffffffff810c2c10>] ? wake_up_state+0x20/0x20 [ 6.568680] [<ffffffffa0321418>] ? __xfs_buf_delwri_submit+0x168/0x250 [xfs] [ 6.576659] [<ffffffffa0321154>] xfs_buf_submit+0x64/0x1c0 [xfs] [ 6.583476] [<ffffffffa0321418>] __xfs_buf_delwri_submit+0x168/0x250 [xfs] [ 6.591260] [<ffffffffa0322141>] ? xfs_buf_delwri_submit+0x41/0xf0 [xfs] [ 6.598852] [<ffffffffa0322141>] xfs_buf_delwri_submit+0x41/0xf0 [xfs] [ 6.606253] [<ffffffffa034d889>] xlog_recover_commit_trans+0x199/0x270 [xfs] [ 6.614234] [<ffffffffa034d9f3>] xlog_recovery_process_trans+0x93/0xd0 [xfs] [ 6.622212] [<ffffffffa034da97>] xlog_recover_process_ophdr+0x67/0xd0 [xfs] [ 6.630093] [<ffffffffa034dba2>] xlog_recover_process_data+0xa2/0xd0 [xfs] [ 6.637875] [<ffffffffa034dc6f>] xlog_recover_process+0x9f/0xf0 [xfs] [ 6.645163] [<ffffffffa034e16d>] xlog_do_recovery_pass+0x4ad/0x620 [xfs] [ 6.652740] [<ffffffffa034e369>] xlog_do_log_recovery+0x89/0xd0 [xfs] [ 6.660037] [<ffffffffa034e3c6>] xlog_do_recover+0x16/0xe0 [xfs] [ 6.666840] [<ffffffffa034f31f>] xlog_recover+0xbf/0x190 [xfs] [ 6.673451] [<ffffffffa03423aa>] xfs_log_mount+0xea/0x2e0 [xfs] [ 6.680168] [<ffffffffa0338c28>] xfs_mountfs+0x518/0x890 [xfs] [ 6.686786] [<ffffffffa0326fd0>] ? xfs_filestream_get_parent+0x80/0x80 [xfs] [ 6.694763] [<ffffffffa0339ad9>] ? xfs_mru_cache_create+0x129/0x190 [xfs] [ 6.702448] [<ffffffffa033c1a3>] xfs_fs_fill_super+0x3b3/0x4d0 [xfs] [ 6.709629] [<ffffffff811fb8d0>] mount_bdev+0x1b0/0x1f0 [ 6.715569] [<ffffffffa033bdf0>] ? xfs_parseargs+0xbe0/0xbe0 [xfs] [ 6.722575] [<ffffffffa033a305>] xfs_fs_mount+0x15/0x20 [xfs] [ 6.729088] [<ffffffff811fc069>] mount_fs+0x39/0x1b0 [ 6.734719] [<ffffffff811a2015>] ? __alloc_percpu+0x15/0x20 [ 6.741029] [<ffffffff81217d1f>] vfs_kern_mount+0x5f/0xf0 [ 6.747154] [<ffffffff8121a27e>] do_mount+0x24e/0xaa0 [ 6.752890] [<ffffffff8119c55b>] ? strndup_user+0x4b/0xa0 [ 6.759011] [<ffffffff8121ab66>] SyS_mount+0x96/0xf0 [ 6.764652] [<ffffffff816954c9>] system_call_fastpath+0x16/0x1b [ 6.771368] XFS (sda9): Metadata corruption detected at xfs_dir3_block_write_verify+0xfd/0x110 [xfs], xfs_dir3_block block 0xfc9f880 [ 6.784666] XFS (sda9): Unmount and run xfs_repair [ 6.790011] XFS (sda9): First 64 bytes of corrupted metadata buffer: [ 6.797104] ffff88086b520000: 58 44 42 33 00 00 00 00 00 00 00 00 0f c9 f8 80 XDB3............ [ 6.806814] ffff88086b520010: 00 00 00 00 00 00 00 00 6e c2 7e d0 a3 ac 4e e7 ........n.~...N. [ 6.816524] ffff88086b520020: 85 5d c8 4c 86 ce f3 a3 00 00 00 00 10 05 36 df .].L..........6. [ 6.826232] ffff88086b520030: 01 08 00 18 02 90 00 18 02 d8 00 18 00 00 00 00 ................ [ 6.835943] XFS (sda9): xfs_do_force_shutdown(0x8) called from line 1288 of file fs/xfs/xfs_buf.c. Return address = 0xffffffffa031f811 [ 6.849531] XFS (sda9): Corruption of in-memory data detected. Shutting down filesystem [ 6.858564] XFS (sda9): Please umount the filesystem and rectify the problem(s) [ 6.920989] XFS (sda9): log mount/recovery failed: error -117 [ 6.927558] XFS (sda9): log mount failed [ FAILED ] Failed to mo[ OK ] Started Emergency Shell. Starting Emergency Shell... [ OK ] Reached target Emergency Mode. Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. :/# [-- MARK -- Fri Jul 15 19:35:00 2016] [-- MARK -- Fri Jul 15 19:40:00 2016] Other job: https://beaker.engineering.redhat.com/recipes/2872992 https://beaker.engineering.redhat.com/recipes/2872993 I will try to reproduce this problem by manual. If I can find root reason. *** This bug has been marked as a duplicate of bug 1375457 *** |
Description of problem: 2nd kernel occur panic when i trigger general kdump testing: [ 3.093155] XFS (dm-0): Internal error XFS_WANT_CORRUPTED_RETURN at line 208 of file fs/xfs/libxfs/xfs_dir2_data.c. Caller xfs_dir3_block_verify+0xd0/0xf0 [ xfs] [ 3.093161] CPU: 0 PID: 708 Comm: mount Not tainted 3.10.0-467.el7.s390x #1 [ 3.093163] 000000007193b2b0 000000007193b340 0000000000000002 0000000 000000000 000000007193b3e0 000000007193b358 000000007193b358 0000000000686e46 0000000000000000 000000000082bd1c 00000000008104a8 000003ff0000000b 000000007193b3a0 000000007193b340 0000000000000000 0000000000000000 000003ff804874e0 00000000001165c8 000000007193b340 000000007193b3a0 [ 3.093173] Call Trace: [ 3.093174] ([<00000000001164a4>] show_trace+0x104/0x158) [ 3.093181] [<000000000011657c>] show_stack+0x84/0x100 [ 3.093182] [<000003ff8041a4b0>] __xfs_dir3_data_check+0x818/0x920 [xfs] [ 3.093195] [<000003ff80417c38>] xfs_dir3_block_verify+0xd0/0xf0 [xfs] [ 3.093208] [<000003ff80417dc0>] xfs_dir3_block_write_verify+0x58/0x120 [xfs ] [ 3.093221] [<000003ff8043a2de>] _xfs_buf_ioapply+0x9e/0x468 [xfs] [ 3.093239] [<000003ff8043a7b8>] xfs_buf_submit+0x88/0x198 [xfs] [ 3.093253] [<000003ff8043aa1e>] __xfs_buf_delwri_submit+0x156/0x258 [xfs] [ 3.093268] [<000003ff8043b954>] xfs_buf_delwri_submit+0x5c/0xc8 [xfs] [ 3.093283] [<000003ff8047008e>] xlog_recover_commit_trans+0x1fe/0x2e8 [xfs] [ 3.093304] [<000003ff8047027c>] xlog_recovery_process_trans+0x104/0x120 [xf s] [ 3.093321] [<000003ff80470346>] xlog_recover_process_ophdr+0xae/0x128 [xfs] [ 3.093337] [<000003ff80470474>] xlog_recover_process_data+0xb4/0x108 [xfs] [ 3.093354] [<000003ff804705da>] xlog_recover_process+0x112/0x170 [xfs] [ 3.093370] [<000003ff80470aa8>] xlog_do_recovery_pass+0x470/0x5f0 [xfs] [ 3.093387] [<000003ff80470cdc>] xlog_do_log_recovery+0xb4/0x110 [xfs] [ 3.093403] [<000003ff80470d78>] xlog_do_recover+0x40/0x140 [xfs] [ 3.093419] [<000003ff80471766>] xlog_recover+0xde/0x1a8 [xfs] [ 3.093437] [<000003ff80461454>] xfs_log_mount+0x114/0x2c8 [xfs] [ 3.093453] [<000003ff80455b9c>] xfs_mountfs+0x48c/0x7d0 [xfs] [ 3.093470] [<000003ff8045a6c6>] xfs_fs_fill_super+0x496/0x5c0 [xfs] [ 3.093486] [<00000000002d6b2a>] mount_bdev+0x1c2/0x1f0 [ 3.093490] [<000003ff80457e3a>] xfs_fs_mount+0x42/0x50 [xfs] [ 3.093506] [<00000000002d75c8>] mount_fs+0x70/0x1a8 [ 3.093508] [<00000000002fb3a6>] vfs_kern_mount+0x7e/0x128 [ 3.093512] [<00000000002fe51a>] do_mount+0x25a/0xa40 [ 3.093514] [<00000000002fedc0>] SyS_mount+0xc0/0x108 [ 3.093516] [<0000000000694154>] sysc_nr_ok+0x22/0x28 [ 3.093520] [<000003fffd387522>] 0x3fffd387522 [ 3.093538] XFS (dm-0): Metadata corruption detected at xfs_dir3_block_write_ verify+0x10e/0x120 [xfs], xfs_dir3_block block 0x1985100 [ 3.093539] XFS (dm-0): Unmount and run xfs_repair [ 3.093540] XFS (dm-0): First 64 bytes of corrupted metadata buffer: [ 3.093543] 0000000070451000: 58 44 42 33 55 fe 3f ba 00 00 00 00 01 98 51 00 XDB3U.?.......Q. [ 3.093544] 0000000070451010: 00 00 00 0b 00 00 2c 18 cc 2c b5 ae ed c2 49 3b ......,..,....I; [ 3.093545] 0000000070451020: b9 f8 de f1 7e 63 18 ee 00 00 00 00 02 0b d8 ac ....~c.......... [ 3.093547] 0000000070451030: 0a 58 03 10 00 a8 00 18 01 08 00 18 00 00 00 20 .X............. [ 3.093549] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1288 of f ile fs/xfs/xfs_buf.c. Return address = 0x000003ff8043a69a [ 3.093550] XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem [ 3.093551] XFS (dm-0): Please umount the filesystem and rectify the problem( s) [ 3.095754] XFS (dm-0): log mount/recovery failed: error -117 [ 3.095833] XFS (dm-0): log mount failed [ [1;31mFAILED [0m] Failed to mount /sysroot. See 'systemctl status sysroot.mount' for details. [ [1;33mDEPEND [0m] Dependency failed for Initrd Root File System. [ [1;33mDEPEND [0m] Dependency failed for Reload Configuration from the Real Roo t. [ [32m OK [0m] Stopped dracut pre-udev hook. Stopping dracut pre-udev hook... [ [32m OK [0m] Stopped dracut cmdline hook. Stopping dracut cmdline hook... [ [32m OK [0m] Stopped dracut initqueue hook. Stopping dracut initqueue hook... [ [32m OK [0m] Reached target Initrd File Systems. [ [32m OK [0m] Stopped target Basic System. [ [32m OK [0m] Stopped target System Initialization. [ [32m OK [0m] Started Emergency Shell. Starting Emergency Shell... [ [32m OK [0m] Reached target Emergency Mode. Generating "/run/initramfs/rdsosreport.txt" kernel-3.10.0-467.el7 kexec-tools-2.0.7-42.el7.x86_64 Version-Release number of selected component (if applicable): How reproducible: 20% Steps to Reproduce: 1. 2. 3. Actual results: Expected results: kdump normal. Additional info: