Bug 182057 - GFS out of memory hang due to large count of files in directory
GFS out of memory hang due to large count of files in directory
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: gfs (Show other bugs)
All Linux
medium Severity medium
: ---
: ---
Assigned To: Wendy Cheng
GFS Bugs
Depends On:
  Show dependency treegraph
Reported: 2006-02-19 16:56 EST by Wendy Cheng
Modified: 2010-01-11 22:09 EST (History)
0 users

See Also:
Fixed In Version: RHBA-2006-0561
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2006-08-10 17:34:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
gfs_malloc_leaf_free.patch (3.09 KB, patch)
2006-02-19 20:15 EST, Wendy Cheng
no flags Details | Diff
gfs_malloc_split.patch (3.53 KB, patch)
2006-02-19 20:16 EST, Wendy Cheng
no flags Details | Diff
gfs_gmalloc_dump.patch (303 bytes, patch)
2006-02-19 20:18 EST, Wendy Cheng
no flags Details | Diff

  None (edit)
Description Wendy Cheng 2006-02-19 16:56:06 EST
Description of problem:
Break out of bugzilla 174576 (RHEL3 INRA customer issue) where GFS could hang 
(looping) due to out of memory errors. Totally 3 patches will be checked into 
CVS RHEL4 using this bugzilla. 

Version-Release number of selected component (if applicable):

How reproducible:
Never tried in house - however, the customer has a directory with rouglhy 
500000 files in it. 

Steps to Reproduce:
Actual results:
GFS hangs.

Expected results:
No hang.

Additional info:
All patches tested out by INRA.
Comment 1 Wendy Cheng 2006-02-19 20:15:15 EST
Created attachment 124873 [details]

Patch 3-1:
Fixes directory delete out of memory error. Found in customer environment
where gfs_inoded is deleting a max size of hash unit (0xffff entries). It
hangs in leaf_free() during gmalloc while kmallocing 0xffff*sizeof(uint64_t)
(=512K) of memroy. It did a kmalloc, zeroed out the buffer, then copied the
zeroed contents into bh buffer and subsequently sent the bh into gfs_writei
to write out to disk. This patch removes the unnecessary kmalloc plus the
memory copy by directly zero out the bh buffer.
Comment 2 Wendy Cheng 2006-02-19 20:16:49 EST
Created attachment 124875 [details]

Patch 3-2:
GFS was trying to split a full-grown directory (0xffff entries) hash leaf
into two and subseqently hang. The buffer requirement 0xffff*sizeof(uint64_t)/2

= 262144 (256K) was too big for kmalloc to handle. Change them into vmalloc if
kmalloc fails.
Comment 3 Wendy Cheng 2006-02-19 20:18:36 EST
Created attachment 124876 [details]

patch 3-3: add dump_stack() into gmalloc so we could know the culprit whenever
out of memory loop occurs.
Comment 6 Red Hat Bugzilla 2006-08-10 17:34:58 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.