Bug 812876 - md-cache: dict caching results in significant memory usage
Summary: md-cache: dict caching results in significant memory usage
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: 2.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Brian Foster
QA Contact: Sachidananda Urs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-04-16 13:15 UTC by Brian Foster
Modified: 2013-09-23 22:32 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.4.0qa4-1.el6rhs
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-09-23 22:32:40 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Brian Foster 2012-04-16 13:15:43 UTC
Description of problem:

Directory recursion causes significant memory usage in the client when the md-cache translator is in use. This seems to occur because md-cache caches a significant amount of dictionary data based on the inode cache, which does not aggressively evict on the client side.

Version-Release number of selected component (if applicable): 3.3.x


How reproducible: 100%


Steps to Reproduce:
1. Create/mount a gluster volume.
2. Copy a relatively large directory tree (i.e., kernel source tree).
3. ls -alR the resulting tree, observe RES/RSS of the glusterfs client process.
4. Replicate the directory tree and repeat step #3.
  
Actual results: With 4 replicated kernel source trees, I reproduce about 8GB of memory used in the client process.


Expected results: Some kind of cap on memory usage.


Additional info: It is expected that md-cache can result in significant memory usage compared to without the translator installed, but the user should be able to restrict the cache from growing beyond a certain point.

Comment 1 Brian Foster 2012-04-16 13:18:19 UTC
I'll try to add an lru mechanism to md-cache (i.e., similar to quick-read) to try and control this behavior...

Comment 2 Brian Foster 2012-04-17 14:31:31 UTC
Avati's comments:

"You make a good observation. The problem I suspect is because of the loaded quick-read translator which results in returning of file data in the lookup_cbk's xattr dict. The md-cache's xattr cache is a superset of which is needed (basically the entire xattr dictionary which also holds quick-read's response and not just the interested keys of md-cache). What is noted might very well be the same overrun behavior of quick-read, and for the same data element (rather than extended attributes). It would be interesting to see if we fix the *xatt_set() in md-cache to selectively dup just the required attributes and cache only those rather than blindly storing the entire dict as-is (and thereby caching file contents which were present to satisfy quick-read's request)."

---

I reproduce a reduction in glusterfs RSS from 28GB to 1.3GB with a particular dataset by removing quick-read from the client graph, which validates the above.

Comment 3 Brian Foster 2012-05-02 13:48:38 UTC
Suggestion from Amar:

< amarts> and about md-cache + quick-read consuming extra memory, we were thinking just doing a 'dict_del(GF_CONTENT_KEY);' in quick-read's lookup_cbk(), before unwind

Comment 4 Brian Foster 2012-05-03 15:06:06 UTC
http://review.gluster.com/3268

Comment 5 Anand Avati 2012-05-08 22:34:09 UTC
CHANGE: http://review.gluster.com/3268 (quick-read, md-cache: selectively cache xattr data to conserve memory) merged in master by Anand Avati (avati@redhat.com)

Comment 7 Sachidananda Urs 2012-12-18 06:50:23 UTC
Tested with more than 10 replicated kernel sources, metadata intensive operations. Memory usage more or less remained constant around 310M.

Couldn't even hit close to 1G. (Earlier hitting around 8G for a couple of replicated kernel sources).

Comment 8 Scott Haines 2013-09-23 22:32:40 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html


Note You need to log in before you can comment on or make changes to this bug.