Bug 1487726
| Summary: | fsck.gfs2 segfaults on corrupt filesystem | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Andreas Gruenbacher <agruenba> | ||||
| Component: | gfs2-utils | Assignee: | Andrew Price <anprice> | ||||
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.4 | CC: | cluster-maint, gfs2-maint, jpayne, rpeterso | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | gfs2-utils-3.1.10-10.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-09-29 20:33:28 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
So the pass2_fxns.check_metalist is NULL:
struct metawalk_fxns pass2_fxns = {
.private = NULL,
.check_leaf_depth = check_leaf_depth,
.check_leaf = NULL,
.check_metalist = NULL,
.check_data = NULL,
.check_eattr_indir = check_eattr_indir,
.check_eattr_leaf = check_eattr_leaf,
.check_dentry = check_dentry,
.check_eattr_entry = NULL,
.check_hash_tbl = check_hash_tbl,
.repair_leaf = pass2_repair_leaf,
};
and in pass2 the root directory inode is getting checked in check_system_dir() using pass2_fxns. The root inode is treated specially as its bitmap entry has been set to BLKST_FREE somehow (part of the corruption) and so check_system_dir() is calling check_metatree() on it:
...
iblock = sysinode->i_di.di_num.no_addr;
ds.q = bitmap_type(sysinode->i_sbd, iblock);
pass2_fxns.private = (void *) &ds;
if (ds.q == GFS2_BLKST_FREE) {
/* First check that the directory's metatree is valid */
error = check_metatree(sysinode, &pass2_fxns);
...
check_metatree() then calls build_and_check_metalist() which calls pass2->check_metalist() and we get:
Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000000040fdf3 in build_and_check_metalist (ip=0xb8d150, mlp=0x7fffffffdbb0, pass=0x64e7e0 <pass2_fxns>) at metawalk.c:1295
#2 0x00000000004108bb in check_metatree (ip=0xb8d150, pass=0x64e7e0 <pass2_fxns>) at metawalk.c:1556
#3 0x000000000042019f in check_system_dir (sysinode=0xb8d150, dirname=0x443acf "root", builder=0x4392f7 <build_root>) at pass2.c:2009
#4 0x0000000000420f07 in pass2 (sdp=0x7fffffffdee0) at pass2.c:2234
#5 0x000000000040c078 in fsck_pass (p=0x43cd80 <passes+32>, sdp=0x7fffffffdee0) at main.c:257
#6 0x000000000040c446 in main (argc=3, argv=0x7fffffffe408) at main.c:335
(gdb) up
#1 0x000000000040fdf3 in build_and_check_metalist (ip=0xb8d150, mlp=0x7fffffffdbb0, pass=0x64e7e0 <pass2_fxns>) at metawalk.c:1295
1295 error = pass->check_metalist(ip, block, &nbh,
(gdb) p pass->check_metalist
$1 = (int (*)(struct gfs2_inode *, uint64_t, struct gfs2_buffer_head **, int, int *, int *, void *)) 0x0
> The root inode is treated specially as its bitmap entry has been set to BLKST_FREE somehow (part of the corruption)
This is incorrect. It seems that the root inode (#33132) is getting marked free by a (weird) previous change that fsck.gfs2 makes:
...
Starting pass1
Inode 33132 (0x816c) has a bad indirect block pointer 61677 (0xf0ed) (points to something that is not a directory hash table block).
Block 61676 (0xf0ec) was 'data', should be free.
The bitmap was fixed.
Error: inode 33132 (0x816c) had unrecoverable errors at metadata block 0 (0x0), offset 0 (0x0), block 0 (0x0).
Undoing metadata work for block 33132 (0x816c)
Block 33132 (0x816c) was 'inode', should be free. <--------
The bitmap was fixed.
The corrupt inode was invalidated.
Inode #33132 (0x816c): Ondisk block count (259) does not match what fsck found (1)
Block count for #33132 (0x816c) fixed
Reconciling bitmaps.
...
It looks like there are two issues here. One is that build_and_check_metalist() calls pass->check_metalist() without checking that it's non-NULL first. The check_metatree() path is already being called for the root directory inode in pass1 so I expect a NULL check before the call is the right way to fix that, rather than adding a ->check_metalist function to pass2_fxns. The other problem is that the root directory inode is getting marked as free after its indirect pointers are found to be bad. My expectation in that case would be for the bad indirect pointers to just get removed, leaving any valid directory entries attached and the directory inode itself in place. Is that what we should be doing instead, perhaps? By the way, if you need a workaround for the corrupt metadata, using gfs2_edit to zero the pointer to block 61677 in the root inode block (33132) will allow fsck.gfs2 to continue and fix the fs. Adding the NULL check fixes the segfault but the fs still needs fixing once fsck.gfs2 has finished. That comes back to the unrecoverable metadata error relating to the root directory inode in pass1: (set_ip_blockmap:1312) directory inode found at block (0x816c): marking as 'inode' (check_metalist:418) inode (0x816c) references indirect block (0xf0ec): marking as 'data' Inode 33132 (0x816c) has a bad indirect block pointer 61677 (0xf0ed) (points to something that is not a directory hash table block). Unrecoverable metadata error on block 61677 (0xf0ed). Further metadata will be skipped. Undoing the work we did before the error on block 33132 (0x816c). (undo_reference:484) inode (0x816c) references bad indirect block (0xf0ec): marking as 'free' Block 61676 (0xf0ec) was 'data', should be free. The bitmap was fixed. (check_metatree:1560) <backtrace> - check_metatree() Error: inode 33132 (0x816c) had unrecoverable errors at metadata block 0 (0x0), offset 0 (0x0), block 0 (0x0). Undoing metadata work for block 33132 (0x816c) (check_metatree:1676) corrupt inode found at block (0x816c): marking as 'free' Block 33132 (0x816c) was 'inode', should be free. The bitmap was fixed. The corrupt inode was invalidated. Inode #33132 (0x816c): Ondisk block count (259) does not match what fsck found (1) inode has: 259, but fsck counts: Dinode:1 + indir:0 + data: 0 + ea: 0 Block count for #33132 (0x816c) fixed So I'll have to figure out what can be done about that. Verified in gfs2-utils-3.1.10-11.el7: [root@host-127 ~]# uname -r 3.10.0-1153.el7.x86_64 [root@host-127 ~]# rpm -q gfs2-utils gfs2-utils-3.1.10-11.el7.x86_64 [root@host-127 ~]# gfs2_edit restoremeta fsck-segfault.gz /dev/sda1 Metadata saved at Fri Sep 1 13:05:22 2017 File system size 63.1023GB Block size is 4096B This is gfs2 metadata. There are 52428790 free blocks on the destination device. Highest saved block is 16711812 (0xff0084) 16711813 blocks processed, 54907 saved (100%) File fsck-segfault.gz restore successful. [root@host-127 ~]# fsck.gfs2 -y /dev/sda1 &> out.txt itializing fsck Validating resource group index. Level 1 resource group check: Checking if all rgrp and rindex values are good. (level 1 passed) Starting pass1 Inode 33132 (0x816c) has a bad indirect block pointer 61677 (0xf0ed) (points to something that is not a directory hash table block). Inode #33132 (0x816c): Ondisk block count (259) does not match what fsck found (66) Block count for #33132 (0x816c) fixed Reconciling bitmaps. Block 33243 (0x81db) bitmap says 1 (data) but FSCK saw 0 (free) Fixed. <=========================SNIPPED====================> Block 61677 (0xf0ed) bitmap says 1 (data) but FSCK saw 0 (free) Fixed. RG #32856 (0x8058) free count inconsistent: is 44548 should be 44741 Resource group counts updated reconcile_bitmaps completed in 0.169s pass1 completed in 0.507s Starting pass1b pass1b completed in 0.000s Starting pass2 Leaf block 46810 (0xb6da) in dinode 33132 (0x816c) has the wrong depth: is 8 (length 4), should be 6 (length 1). The leaf block depth was fixed. <================================SNIPPED=============================> Added inode #61641 (0xf0c9) to lost+found Found unlinked inode at 61643 (0xf0cb) Added inode #61643 (0xf0cb) to lost+found Found unlinked inode at 61644 (0xf0cc) Unlinked inode has zero size Block 61644 (0xf0cc) was 'inode', should be free. The bitmap was fixed. Found unlinked inode at 61678 (0xf0ee) Added inode #61678 (0xf0ee) to lost+found pass4 completed in 0.436s Starting check_statfs The statfs file is wrong: Current statfs values: blocks: 16775236 (0xfff844) free: 16721550 (0xff268e) dinodes: 20332 (0x4f6c) Calculated statfs values: blocks: 16775236 (0xfff844) free: 16721432 (0xff2618) dinodes: 20333 (0x4f6d) The statfs file was fixed. check_statfs completed in 0.000s Writing changes to disk gfs2_fsck complete Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (gfs2-utils bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4008 |
Created attachment 1321082 [details] Filesystem metadata Description of problem: Running "fsck.gfs2 -y <device>" segfaults on a corrupt filesystem (metadata attached). The corruption resulted from a kernel bug we haven't tracked down, yet; the filesystem seems to be out of inodes. Version-Release number of selected component (if applicable): gfs2-utils from rhel7 up to current master branch (3.1.10-14-g4cb82588). How reproducible: Always. Steps to Reproduce: 1. lvcreate -n fsck -L 64G vg0 2. gfs2_edit restoremeta fsck-segfault.gz /dev/mapper/vg0-fsck 3. fsck.gfs2 -y /dev/mapper/vg0-fsck Actual results: [...] Starting pass2 Segmentation fault (core dumped) Expected results: No segfault, repaired filesystem.