Bug 442271 - GFS: gfs_fsck bugs found in rindex repair code
GFS: gfs_fsck bugs found in rindex repair code
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: gfs-utils (Show other bugs)
5.3
All Linux
low Severity low
: rc
: ---
Assigned To: Robert Peterson
GFS Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-13 13:10 EDT by Robert Peterson
Modified: 2010-01-11 22:34 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-02 07:01:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Proposed Patch (8.20 KB, patch)
2008-04-14 15:34 EDT, Robert Peterson
no flags Details | Diff
Latest patch (13.09 KB, patch)
2008-09-24 10:41 EDT, Robert Peterson
no flags Details | Diff

  None (edit)
Description Robert Peterson 2008-04-13 13:10:54 EDT
Description of problem:
When I was working on bug #440897, I got some user metadata in,
including a partial rindex file.  I had hoped I could use gfs_fsck
to fill in the gaps and repair my rindex.  Instead, I uncovered
some bugs in the rindex repair code.  This bug will be used to
fix those bugs.

Version-Release number of selected component (if applicable):
RHEL5.3 prototype

How reproducible:
Always

Steps to Reproduce:
1. Blast the last couple blocks of a non-trivial rindex file
2. Run gfs_fsck
  
Actual results:
gfs_fsck is unable to repair the rindex, segfaults

Expected results:
gfs_fsck should be able to fix the problem.

Additional info:
I've got a prototype that passes my gfs_fsck_hellfire test.
Comment 2 Robert Peterson 2008-04-14 15:34:22 EDT
Created attachment 302383 [details]
Proposed Patch

This is the first prototype I wrote about in the description.  It
passes my "gfs_fsck_hellfire" test.  I should test it on some more
shredded rindex files though.
Comment 3 RHEL Product and Program Management 2008-06-04 18:45:16 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 4 Robert Peterson 2008-06-26 14:44:09 EDT
I got some customer feedback that indicated it would be worth adding
a special parameter to gfs_fsck to force a rindex rebuild.  As it stands
today, it doesn't always detect some kinds of rindex corruption.
Comment 5 Robert Peterson 2008-07-10 12:00:43 EDT
Note that since "normal" gfs2's RGs are always the same size, even 
after a gfs2_grow operation, we don't need to port this patch to
gfs2_fsck.  However, this does expose a bigger issue: file systems
that are converted from gfs to gfs2 via gfs2_convert will have their
RGs in the same locations as they were in gfs.  Today's rindex and
rgrp repair algorithm in gfs2_fsck are grossly inadequate to handle
that situation because gfs's RG repair algorithm is very involved and
complex.  The gfs2 version is much simplified and cleaner, relying
on the fact that RGs will always be on nice neat boundaries.

Seeing as how rindex and rgrp corruption are very rare to begin with,
there should be minimal customer exposure.  However, if need be we can
lift the entire rg repair algorithm from gfs_fsck and make a special
repair case for file systems converted from gfs.  That port might be
a big effort for very little payback.  Still, the first customer to
have RG damage on a gfs1-converted file system won't be in a very good
mood.  One saving grace is that the instructions in gfs2_convert tell
the customers to run gfs_fsck before converting the file system, so
hopefully RG and rindex damage won't be carried over across the convert.
So in theory the problem should only be for RG damage that occurs
after the conversion takes place.  Time will tell if it's worth doing.
Comment 7 Robert Peterson 2008-07-24 12:39:46 EDT
If and when I do this fix, I should crosswrite patch 768d7f6 from gfs2
to gfs.  This is the problem where RG blocks inside a journal look just
like a real RG block and confuse the rg repair algorithm, making it
think the RGs have improper sizes.  In theory, this should cause
gfs_fsck to determine improper block locations for the RGs, and then,
after discovering those blocks aren't really RGs, it quits with
"Error: too many bad RGs."
Comment 8 Robert Peterson 2008-09-24 10:41:25 EDT
Created attachment 317593 [details]
Latest patch

This patch fixes some problems uncovered by a destroyed rindex file
that came in from a user.  With this patch, he was able to recover
his 2TB gfs file system successfully, and the rindex was very badly
destroyed.  I still need to run this version through some tests but
this is what I'm planning to ship unless I find bugs.
Comment 10 Robert Peterson 2008-12-22 16:33:57 EST
Equivalent patches were pushed to the master, STABLE2, STABLE3 and
RHEL5 branches of the cluster git tree for inclusion into 5.4.
RHEL5 patch was tested on system roth-01 with a variety of scenarios
including my gfs_fsck_hellfire test case.  I also tested the patch
against some user metadata from previous bugs.  Changing status to
Modified.
Comment 11 Fedora Update System 2009-01-23 21:36:12 EST
gfs2-utils-2.03.11-1.fc9, cman-2.03.11-1.fc9, rgmanager-2.03.11-1.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 13 Jaroslav Kortus 2009-06-11 06:03:33 EDT
There is different behaviour of the tool on x86_64 and on ia64. On x86_64 it works as expected but does not fix the FS on ia64. Tested with GFS fsck 0.1.19 (built May  4 2009 19:35:05). Similar corruption (rindex entry corruption, part of gfs_fsck_hellfire) in gfs2 ends in segfault on both archs.

This is not regression as I think the fix never really worked on ia64 (tested previous 5.x releases).

Example of successful run on x86_64:
x86_64 - rindex entry #1 -> 0x00
gfs_fsck -y /dev/VolGroup00/GFS
Initializing fsck
Invalid length 0 found in rindex.
number_of_rgs = 8.
rgindex #1 ri_addr discrepancy: index 0x0 != expected: 0x11
rgindex #1 ri_length discrepancy: index 0x0 != expected: 0x4
rgindex #1 ri_data1 discrepancy: index 0x0 != expected: 0x15
rgindex #1 ri_data discrepancy: index 0x0 != expected: 0xebf8
rgindex #1 ri_bitbytes discrepancy: index 0x0 != expected: 0x3afe
Clearing journals (this may take a while).
Journals cleared.
Starting pass1
Pass1 complete      
Starting pass1b
Pass1b complete      
Starting pass1c
Pass1c complete      
Starting pass2
Pass2 complete      
Starting pass3
Pass3 complete      
Starting pass4
Pass4 complete      
Starting pass5
Pass5 complete      
Writing changes to disk

Note the rgindex #1 (the only one corrupted) is fixed with correct values.

Now ia64 the same scenario:
ia64 - rindex entry #1 -> 0x00 (repeats, uncorrectable)

gfs_fsck -y /dev/sdc1
Initializing fsck
Invalid length 0 found in rindex.
The middle RG is not on an even boundary (fs has grown?)
Section 1: 0x11 - 0x1bcdf
  RG 1 at block 0x11 intact [length 0x6f33]
  RG 2 at block 0x6F44 intact [length 0x6f30]
  RG 3 at block 0xDE74 intact [length 0x6f30]
  RG 4 at block 0x14DA4 intact [length 0x6f30]
  RG 5 at block 0x1BCD4 intact [length 0x6f30]
Section 2: 0x1fcc0 - 0x3b99f
* RG 6 at block 0x1FCC0 *** DAMAGED *** [length 0x6f4c]
  RG 7 at block 0x26C0C intact [length 0x6f34]
  RG 8 at block 0x2DB40 intact [length 0x6f34]
  RG 9 at block 0x34A74 intact [length 0x6f34]
Section 3: 0x3b9a0 - 0x3b9a7

Unable to use rindex; doing block-by-block search.
This will be slow, so be patient.
* RG 10 at block 0x3B9A0 *** DAMAGED *** [length 0x1]
rgindex #1 ri_addr discrepancy: index 0x0 != expected: 0x11
rgindex #1 ri_length discrepancy: index 0x0 != expected: 0x2
rgindex #1 ri_data1 discrepancy: index 0x0 != expected: 0x13
rgindex #1 ri_data discrepancy: index 0x0 != expected: 0x6f30
rgindex #1 ri_bitbytes discrepancy: index 0x0 != expected: 0x1bcc
rgindex #5 ri_addr discrepancy: index 0x1fcd5 != expected: 0x1bcd4
rgindex #5 ri_length discrepancy: index 0x2 != expected: 0x1
rgindex #5 ri_data1 discrepancy: index 0x1fcd7 != expected: 0x1bcd5
rgindex #5 ri_data discrepancy: index 0x6f34 != expected: 0x0
rgindex #5 ri_bitbytes discrepancy: index 0x1bcd != expected: 0x0
rgindex #6 ri_addr discrepancy: index 0x26c0c != expected: 0x1fcc0
rgindex #6 ri_length discrepancy: index 0x2 != expected: 0x1
rgindex #6 ri_data1 discrepancy: index 0x26c0e != expected: 0x1fcc1
rgindex #6 ri_data discrepancy: index 0x6f30 != expected: 0x6f48
rgindex #6 ri_bitbytes discrepancy: index 0x1bcc != expected: 0x1bd2
rgindex #7 ri_addr discrepancy: index 0x2db40 != expected: 0x26c0c
rgindex #7 ri_data1 discrepancy: index 0x2db42 != expected: 0x26c0e
rgindex #8 ri_addr discrepancy: index 0x34a74 != expected: 0x2db40
rgindex #8 ri_data1 discrepancy: index 0x34a76 != expected: 0x2db42
Resource group count discrepancy. Index says 8. Should be 10.
Block #130240 (0x1fcc0) (1 of 1) is neither GFS_METATYPE_RB nor GFS_METATYPE_RG.
Attempting to repair the RG.
Clearing journals (this may take a while).
Journals cleared.
Starting pass1
Block #130240 (0x1fcc0) (1 of 1) is neither GFS_METATYPE_RB nor GFS_METATYPE_RG.
Resource group or index is corrupted.


Several non-existing errors in FS are discovered. Please note that the disk was zeroed before mkfs_gfs (mkfs.gfs -O -t a3cluster:a3gfs2 -p lock_nolock -j 2 -J 32 /dev/sdc1). The errors repeat each run and from 2nd on are always the same. I was unable to fix the FS on ia64 with gfs_fsck.

And last, example of backtrace of gfs2_fsck after zeroing first rgindex:
ia64 coredump:
Core was generated by `gfs2_fsck -y /dev/sdc1'.
Program terminated with signal 11, Segmentation fault.
[New process 9558]
#0  0x4000000000049fe0 in gfs2_rgrp_read (sdp=0x60000fffff879220, rgd=0x6000000000057290) at rgrp.c:148
148             gfs2_rgrp_in(&rgd->rg, rgd->bh[0]->b_data);
(gdb) bt full
#0  0x4000000000049fe0 in gfs2_rgrp_read (sdp=0x60000fffff879220, rgd=0x6000000000057290) at rgrp.c:148
        x = 0
        length = 0
#1  0x40000000000185a0 in rg_repair ()
No symbol table info available.
#2  0x4000000000005580 in initialize ()
No symbol table info available.
#3  0x4000000000003170 in main ()
No symbol table info available.
(gdb)
Comment 14 Jaroslav Kortus 2009-07-20 11:58:42 EDT
verified with gfs-utils-0.1.20-1.el5.
The fix is working apart from setup described in bug 512722.
Comment 17 errata-xmlrpc 2009-09-02 07:01:21 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1336.html

Note You need to log in before you can comment on or make changes to this bug.