Red Hat Bugzilla – Bug 445858
GFS: gfs_fsck cannot allocate enough memory to run on large file systems
Last modified: 2010-03-15 13:14:54 EDT
Description of problem:
When running on RHEl5 on a X86 32bit system, gfs_fsck can not allocate enough
memory to run when the file system size is very large.
Steps to Reproduce:
1. Create a GFS 16TB file system
2. Perform a gfs_fsck
gfs_fsck -vvv -n /dev/backup/wav
Initializing special inodes...
(file.c:45) readi: Offset (400) is >= the file size (400).
(super.c:226) 5 journals found.
Validating Resource Group index.
Level 1 check.
(file.c:45) readi: Offset (1468608) is >= the file size (1468608).
15298 resource groups found.
Setting block ranges...
Creating a block list of size 4160749568...
(bitmap.c:68) Allocated bitmap of size 2080374785 with 2 chunks per byte
(bitmap.c:68) Allocated bitmap of size 520093697 with 8 chunks per byte
Unable to allocate bitmap of size 520093697
(block_list.c:72) <backtrace> - block_list_create()
This system doesn't have enough memory + swap space to fsck this file system.
Additional memory needed is approximately: 5952MB
Please increase your swap space by that amount and run gfs_fsck again.
(initialize.c:400) <backtrace> - init_sbp()
The gfs_fsck should complete normally.
Reassigning to myself: I've been working with Ben on this.
According to http://www.redhat.com/rhel/compare/ the RHEL5 release
does not ship with a HUGEMEM kernel, so on x86 (32-bit) platforms, the
system is limited to 3GB of address space. So regardless of how much
swap and/or ram you have, gfs_fsck cannot run properly today on a
16TB file system as documented, even if gfs itself can run.
It turns out that for a 16TB file system, gfs_fsck allocates a 2GB chunk
of RAM, then 3 more 1G chunks of RAM, for its internal bitmaps. These
bitmaps are needed to keep track of every block in the file system and
to determine what kind of block it is (inode, directory, data, duplicate,
One way I've thought of to fix it is to try to process the bitmaps
one resource group at a time, rather than all bitmaps in memory at once.
The problem with that approach is that a block in one resource group can
reference blocks in another resource group. So it might get complex
trying to keep the cross-RG references straight. Plus each pass the
code makes references the bits left behind by the previous passes.
Another thought is to try to pare down the memory usage by combining
certain bits in the bitmaps and adding some code to resolve the type.
Perhaps we can get it down to one bitmap of 2G that way.
Right now, the only circumvention is to do the gfs_fsck on a 64-bit arch.
Correction to comment #1: It allocates 2G for bitmaps, then three
smaller bitmaps of 512MB each (not 1G as previous stated).
I've been working on the issue of gfs_fsck memory usage indirectly
because gfs2_fsck has the same problem. For bug #404611, I wanted
to test a gfs2_fsck against a 2T file system that's had millions of
files and directories created through a benchmark program called benchp.
The test system, kool, has 2GB of memory.
The test didn't go so well because of gfs2_fsck's memory problems that
are directly inherited from gfs_fsck. I created a patch that saves
lots of memory. Basically, I eliminated three of the four bitmaps
associated with the file system and made some data structures smaller.
See the attachment associated with this comment:
Despite the patch, the program ran all weekend and only made it to 3%
done with pass1. In doing all this, I've researched where gfs_fsck and
gfs2_fsck are using their memory. A big part of the problem is that
during pass1, it creates elements in memory associated with every inode
and every directory. Their purpose is to keep counts, primarily the
count of links, to be used later in pass4.
So the patch is not enough memory savings. In fact, the cpu is way
underutilized and the system spends most of its time swapping to disk.
So I'm looking for more good ways to improve the memory usage and
anything I find in gfs2_fsck should directly apply to gfs_fsck.
The fix isn't ready to ship yet; retargeting to 5.6.
I don't think we should fix this bug for several reasons:
(1) It would be a lot of work and redesign for gfs's fsck.
(2) gfs is being phased out in favor of gfs2, where we should
consider fixing it.
(3) To the best of my knowledge, no customers have expressed
an interest in the fix.
(4) There are no customer issues attached to the bug.
I think the best course of action is to close this as WONTFIX
unless customers start complaining and demanding a fix.
Until that time, I think I'll open a DOC bug to document the
approximate memory requirements for gfs_fsck.
In gfs_fsck the majority of the memory used is consumed for
the block maps that are all kept in memory. There is one big
array that needs a nibble (half-byte) for each block, and
three smaller arrays that need two bits per block. Add to
that the additional memory needed for the buffers, the dinode
hash table, the directory hash table and the duplicates linked
list. So every block needs at least 7 bits plus slop, so call
it 8 bits, or one byte, per block. So a good estimate is
file system size (in bytes) divided by the block size, and that
will be approximately how much memory you will need to run
For this particular 16TB file system, the file system is 16TB
and the block size is 4K, so 16TB / 4K blocks:
17592186044416 / 4096 = 4294967296, so this file system
requires approximately 4GB of free memory to run gfs_fsck,
above and beyond all the memory used for the operating system
Note that if the block size was 1K, it would require four times