Red Hat Bugzilla – Bug 509225
GFS: gfs_fsck sometimes needs to be run twice
Last modified: 2014-06-09 08:23:54 EDT
Created attachment 355278 [details]
Performance patch for STABLE3
In testing this bug, 509225, I ran fsck.gfs on a set of saved GFS
metadata that was 2TB with lots of data. It took 49 hours to run.
For the record, this was "west-0406.img" from our QE group, the
NESW cluster, contributed by Nate Straz (thanks Nate!).
Although the test was successful, I became annoyed that gfs_fsck
was taking so long to run. After all, it only took five minutes
to restore the metadata. The top command indicated it was CPU
bound, which seemed absurd; in an ideal world, fsck should be
disk bound, not cpu bound. The good news is that it only used
one fourth of the available memory (500MB out of 2GB: I expected
I stopped the code with the gdb debugger and looked at where it
was spending its time; not as scientific as using a tool like
gprof, but I had already started the test and didn't want to
have to restart it after so much time had been invested.
Based on what I saw, I coded up a set of simple-but-fsck-pervasive
performance improvements. I implemented three basic performance
ideas: (1) I back-ported the bitmap performance improvements I
had done a long time ago for gfs2's fsck. (2) I optimized some
of the code that is called for every block, such as the bitmap
functions, getting rid of unused parameters, simplifying data
structures and such. (3) I determined that an internal type,
"journal_blk" was used nearly identically to another type,
"meta_other". By combining the two, I was able to re-use the
bit designation for another purpose: "bad_block." That enabled
me to eliminate an entire specialty bitmap (which buys us back
a significant amount of memory), and eliminated a lot of code
needed to handle the out-of-regular-bitmap "bad-block" cases.
I re-ran my test with Nate's file system this weekend and was
pleasantly surprised: This performance patch made fsck.gfs
nearly four times faster, and uses 7/8 of the memory (saves 1/8
in memory requirements). So my fsck.gfs test on Nate's file
system went from 49 hours down to 12.7 hours.
The performance improvement is probably out of the scope of
this bug, and has not yet been cross-written to the RHEL version
of fsck.gfs. So I will likely ship the 509225 without the
performance patch and open a new bug to try to get it into 5.5.
I forgot about a fourth performance improvement included in the
patch above. I had noticed that the code was often re-reading and
re-checking the rgrp for every block. So I added code to keep
the previous rgrp around through every pass of the inner loop when
checking metadata. That was if you have a really big file, with
many consecutive data blocks from the same rgrp, we save a lot of
time and effort re-reading and verifying the rgrp for each block.
Created attachment 355686 [details]
Latest and greatest RHEL5 patch
Created attachment 356610 [details]
This patch fixes an additional problem discovered by a customer
who kept getting this message:
Directory block X, entry 1 of directory Y is corrupt.
I tested it on the customer's metadata and it works correctly
the first time. Due to the change, I need to re-test against
my other metadata again.
Created attachment 356684 [details]
Hopefully the final version of the patch
This patch corrects a few minor things I found while doing
cross-checking and cross-testing with GFS2's fsck for bug #500483.
Created attachment 356892 [details]
Same patch, rebased to current RHEL5 tree
This is the same patch, but it's been rebased to the most recent
RHEL5 git tree. That means it should apply cleanly for people
on newer versions of the git repo.
Requesting ACK flags for 5.5
I've pushed this patch to the master branch of the gfs1-utils
git tree, and the STABLE3 and STABLE2 branches of the cluster
git tree. I'm holding off on the RHEL5 branch until I can get
the rest of the ack flags.
The patch is now pushed to the RHEL5 branch of the cluster.git tree
for inclusion into 5.5. It was tested on systems roth-01 and kool.
Changing status to Modified.
Pushed to the RHEL55 branch of cluster.git. Changing status to POST.
Build completed successfully, changing status to Modified.
A regression was found by the upstream community (bug #521068).
I already have a fix, so I just need to respin the patch.
Changing status to FAILS_QA until that's done.
The addendum patch has been pushed to the RHEL55 and STABLE2
branches of git. The build is complete and successful. Changing
status back to Modified.
~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~
RHEL 5.5 Beta has been released! There should be a fix present in this
release that addresses your request. Please test and report back results
here, by March 3rd 2010 (2010-03-03) or sooner.
Upon successful verification of this request, post your results and update
the Verified field in Bugzilla with the appropriate value.
If you encounter any issues while testing, please describe them and set
this bug into NEED_INFO. If you encounter new defects or have additional
patch(es) to request for inclusion, please clone this bug per each request
and escalate through your support representative.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.