Bug 509225 - GFS: gfs_fsck sometimes needs to be run twice
GFS: gfs_fsck sometimes needs to be run twice
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: gfs-utils (Show other bugs)
All Linux
medium Severity medium
: rc
: ---
Assigned To: Robert Peterson
Cluster QE
Depends On:
  Show dependency treegraph
Reported: 2009-07-01 15:58 EDT by Robert Peterson
Modified: 2014-06-09 08:23 EDT (History)
8 users (show)

See Also:
Fixed In Version: gfs-utils-0.1.20-5.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 500483
Last Closed: 2010-03-30 04:55:44 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Performance patch for STABLE3 (57.92 KB, patch)
2009-07-27 10:09 EDT, Robert Peterson
no flags Details | Diff
Latest and greatest RHEL5 patch (99.26 KB, patch)
2009-07-30 10:43 EDT, Robert Peterson
no flags Details | Diff
Today's patch (105.26 KB, patch)
2009-08-06 23:35 EDT, Robert Peterson
no flags Details | Diff
Hopefully the final version of the patch (106.39 KB, patch)
2009-08-07 12:53 EDT, Robert Peterson
no flags Details | Diff
Same patch, rebased to current RHEL5 tree (105.99 KB, patch)
2009-08-10 09:48 EDT, Robert Peterson
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0290 normal SHIPPED_LIVE gfs-utils bug fix update 2010-03-29 10:09:53 EDT

  None (edit)
Comment 1 Robert Peterson 2009-07-27 10:09:48 EDT
Created attachment 355278 [details]
Performance patch for STABLE3

In testing this bug, 509225, I ran fsck.gfs on a set of saved GFS
metadata that was 2TB with lots of data.  It took 49 hours to run.
For the record, this was "west-0406.img" from our QE group, the
NESW cluster, contributed by Nate Straz (thanks Nate!).

Although the test was successful, I became annoyed that gfs_fsck
was taking so long to run.  After all, it only took five minutes
to restore the metadata.  The top command indicated it was CPU
bound, which seemed absurd; in an ideal world, fsck should be
disk bound, not cpu bound.  The good news is that it only used
one fourth of the available memory (500MB out of 2GB: I expected

I stopped the code with the gdb debugger and looked at where it
was spending its time; not as scientific as using a tool like
gprof, but I had already started the test and didn't want to
have to restart it after so much time had been invested.

Based on what I saw, I coded up a set of simple-but-fsck-pervasive
performance improvements.  I implemented three basic performance
ideas:  (1) I back-ported the bitmap performance improvements I
had done a long time ago for gfs2's fsck.  (2) I optimized some
of the code that is called for every block, such as the bitmap
functions, getting rid of unused parameters, simplifying data
structures and such.  (3) I determined that an internal type,
"journal_blk" was used nearly identically to another type,
"meta_other".  By combining the two, I was able to re-use the
bit designation for another purpose: "bad_block."  That enabled
me to eliminate an entire specialty bitmap (which buys us back
a significant amount of memory), and eliminated a lot of code
needed to handle the out-of-regular-bitmap "bad-block" cases.

I re-ran my test with Nate's file system this weekend and was
pleasantly surprised:  This performance patch made fsck.gfs
nearly four times faster, and uses 7/8 of the memory (saves 1/8
in memory requirements).  So my fsck.gfs test on Nate's file
system went from 49 hours down to 12.7 hours.

The performance improvement is probably out of the scope of
this bug, and has not yet been cross-written to the RHEL version
of fsck.gfs.  So I will likely ship the 509225 without the
performance patch and open a new bug to try to get it into 5.5.
Comment 2 Robert Peterson 2009-07-27 10:28:45 EDT
I forgot about a fourth performance improvement included in the
patch above.  I had noticed that the code was often re-reading and
re-checking the rgrp for every block.  So I added code to keep
the previous rgrp around through every pass of the inner loop when
checking metadata.  That was if you have a really big file, with
many consecutive data blocks from the same rgrp, we save a lot of
time and effort re-reading and verifying the rgrp for each block.
Comment 3 Robert Peterson 2009-07-30 10:43:35 EDT
Created attachment 355686 [details]
Latest and greatest RHEL5 patch
Comment 5 Robert Peterson 2009-08-06 23:35:05 EDT
Created attachment 356610 [details]
Today's patch

This patch fixes an additional problem discovered by a customer
who kept getting this message:

Directory block X, entry 1 of directory Y is corrupt.

I tested it on the customer's metadata and it works correctly
the first time.  Due to the change, I need to re-test against
my other metadata again.
Comment 6 Robert Peterson 2009-08-07 12:53:01 EDT
Created attachment 356684 [details]
Hopefully the final version of the patch

This patch corrects a few minor things I found while doing
cross-checking and cross-testing with GFS2's fsck for bug #500483.
Comment 7 Robert Peterson 2009-08-10 09:48:27 EDT
Created attachment 356892 [details]
Same patch, rebased to current RHEL5 tree

This is the same patch, but it's been rebased to the most recent
RHEL5 git tree.  That means it should apply cleanly for people
on newer versions of the git repo.
Comment 8 Robert Peterson 2009-08-10 15:10:22 EDT
Requesting ACK flags for 5.5
Comment 9 Robert Peterson 2009-08-10 15:27:40 EDT
I've pushed this patch to the master branch of the gfs1-utils
git tree, and the STABLE3 and STABLE2 branches of the cluster
git tree.  I'm holding off on the RHEL5 branch until I can get
the rest of the ack flags.
Comment 10 Robert Peterson 2009-08-10 16:57:36 EDT
The patch is now pushed to the RHEL5 branch of the cluster.git tree
for inclusion into 5.5.  It was tested on systems roth-01 and kool.
Changing status to Modified.
Comment 11 Robert Peterson 2009-08-17 12:23:22 EDT
Pushed to the RHEL55 branch of cluster.git.  Changing status to POST.
Comment 12 Robert Peterson 2009-08-18 11:59:26 EDT
Build completed successfully, changing status to Modified.
Comment 13 Robert Peterson 2009-09-08 14:05:41 EDT
A regression was found by the upstream community (bug #521068).
I already have a fix, so I just need to respin the patch.
Changing status to FAILS_QA until that's done.
Comment 14 Robert Peterson 2009-09-08 15:44:01 EDT
The addendum patch has been pushed to the RHEL55 and STABLE2
branches of git.  The build is complete and successful.  Changing
status back to Modified.
Comment 17 Chris Ward 2010-02-11 05:29:35 EST
~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~

RHEL 5.5 Beta has been released! There should be a fix present in this 
release that addresses your request. Please test and report back results 
here, by March 3rd 2010 (2010-03-03) or sooner.

Upon successful verification of this request, post your results and update 
the Verified field in Bugzilla with the appropriate value.

If you encounter any issues while testing, please describe them and set 
this bug into NEED_INFO. If you encounter new defects or have additional 
patch(es) to request for inclusion, please clone this bug per each request
and escalate through your support representative.
Comment 20 errata-xmlrpc 2010-03-30 04:55:44 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.