Escalated to Bugzilla from IssueTracker
Created attachment 126424 [details] sysreport from cluster machine
I was able to recreate this failure using a sparse device and gdb. The problem should be easy to figure out and, with a little luck, easy to fix.
Created attachment 126571 [details] Patch to fix the problem This problem occurs if gfs_fsck is run on a filesystem that is more than 5TB in size, and possibly under other conditions. What really triggers it is when the hidden/internal resource group index file (rgindex) has more than one level of indirection. In my recreation, the rgindex had 20888 resource group entries. While gfs_fsck is running, it changes the lock protocol to ensure that noone can mount the filesystem as it is being checked. Once the error has occurred and gfs_fsck has given up on a filesystem, it exits, but forgets to change the lock protocol back to lock_dlm, thus making the filesystem unusable. To make the filesystem usable again, merely change the protocol back to lock_dlm as follows: gfs_tool sb /dev/<your device> proto lock_dlm The problem described by this bug was caused by an omission from gfs_fsck's bitmap code that appears to be complete in the gfs kernel code. This patch attempts to fix the problem. It's been tested and works on a 5.1TB sparse device filesystem (basically, faking out the device mapper to think the filesystem is bigger than it really is). The fix is basically to handle the level of indirection needed rather than exiting.
*** Bug 191708 has been marked as a duplicate of this bug. ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2006-0560.html