Created attachment 355278 [details] Performance patch for STABLE3 In testing this bug, 509225, I ran fsck.gfs on a set of saved GFS metadata that was 2TB with lots of data. It took 49 hours to run. For the record, this was "west-0406.img" from our QE group, the NESW cluster, contributed by Nate Straz (thanks Nate!). Although the test was successful, I became annoyed that gfs_fsck was taking so long to run. After all, it only took five minutes to restore the metadata. The top command indicated it was CPU bound, which seemed absurd; in an ideal world, fsck should be disk bound, not cpu bound. The good news is that it only used one fourth of the available memory (500MB out of 2GB: I expected more). I stopped the code with the gdb debugger and looked at where it was spending its time; not as scientific as using a tool like gprof, but I had already started the test and didn't want to have to restart it after so much time had been invested. Based on what I saw, I coded up a set of simple-but-fsck-pervasive performance improvements. I implemented three basic performance ideas: (1) I back-ported the bitmap performance improvements I had done a long time ago for gfs2's fsck. (2) I optimized some of the code that is called for every block, such as the bitmap functions, getting rid of unused parameters, simplifying data structures and such. (3) I determined that an internal type, "journal_blk" was used nearly identically to another type, "meta_other". By combining the two, I was able to re-use the bit designation for another purpose: "bad_block." That enabled me to eliminate an entire specialty bitmap (which buys us back a significant amount of memory), and eliminated a lot of code needed to handle the out-of-regular-bitmap "bad-block" cases. I re-ran my test with Nate's file system this weekend and was pleasantly surprised: This performance patch made fsck.gfs nearly four times faster, and uses 7/8 of the memory (saves 1/8 in memory requirements). So my fsck.gfs test on Nate's file system went from 49 hours down to 12.7 hours. The performance improvement is probably out of the scope of this bug, and has not yet been cross-written to the RHEL version of fsck.gfs. So I will likely ship the 509225 without the performance patch and open a new bug to try to get it into 5.5.
I forgot about a fourth performance improvement included in the patch above. I had noticed that the code was often re-reading and re-checking the rgrp for every block. So I added code to keep the previous rgrp around through every pass of the inner loop when checking metadata. That was if you have a really big file, with many consecutive data blocks from the same rgrp, we save a lot of time and effort re-reading and verifying the rgrp for each block.
Created attachment 355686 [details] Latest and greatest RHEL5 patch
Created attachment 356610 [details] Today's patch This patch fixes an additional problem discovered by a customer who kept getting this message: Directory block X, entry 1 of directory Y is corrupt. I tested it on the customer's metadata and it works correctly the first time. Due to the change, I need to re-test against my other metadata again.
Created attachment 356684 [details] Hopefully the final version of the patch This patch corrects a few minor things I found while doing cross-checking and cross-testing with GFS2's fsck for bug #500483.
Created attachment 356892 [details] Same patch, rebased to current RHEL5 tree This is the same patch, but it's been rebased to the most recent RHEL5 git tree. That means it should apply cleanly for people on newer versions of the git repo.
Requesting ACK flags for 5.5
I've pushed this patch to the master branch of the gfs1-utils git tree, and the STABLE3 and STABLE2 branches of the cluster git tree. I'm holding off on the RHEL5 branch until I can get the rest of the ack flags.
The patch is now pushed to the RHEL5 branch of the cluster.git tree for inclusion into 5.5. It was tested on systems roth-01 and kool. Changing status to Modified.
Pushed to the RHEL55 branch of cluster.git. Changing status to POST.
Build completed successfully, changing status to Modified.
A regression was found by the upstream community (bug #521068). I already have a fix, so I just need to respin the patch. Changing status to FAILS_QA until that's done.
The addendum patch has been pushed to the RHEL55 and STABLE2 branches of git. The build is complete and successful. Changing status back to Modified.
~~ Attention Customers and Partners - RHEL 5.5 Beta is now available on RHN ~~ RHEL 5.5 Beta has been released! There should be a fix present in this release that addresses your request. Please test and report back results here, by March 3rd 2010 (2010-03-03) or sooner. Upon successful verification of this request, post your results and update the Verified field in Bugzilla with the appropriate value. If you encounter any issues while testing, please describe them and set this bug into NEED_INFO. If you encounter new defects or have additional patch(es) to request for inclusion, please clone this bug per each request and escalate through your support representative.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0290.html