Bug 812876

Summary:	md-cache: dict caching results in significant memory usage
Product:	Red Hat Gluster Storage	Reporter:	Brian Foster <bfoster>
Component:	glusterfs	Assignee:	Brian Foster <bfoster>
Status:	CLOSED ERRATA	QA Contact:	Sachidananda Urs <surs>
Severity:	high	Docs Contact:
Priority:	medium
Version:	2.0	CC:	amarts, gluster-bugs, jdarcy, sac, sdharane, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.4.0qa4-1.el6rhs	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-09-23 22:32:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Brian Foster 2012-04-16 13:15:43 UTC

Description of problem:

Directory recursion causes significant memory usage in the client when the md-cache translator is in use. This seems to occur because md-cache caches a significant amount of dictionary data based on the inode cache, which does not aggressively evict on the client side.

Version-Release number of selected component (if applicable): 3.3.x


How reproducible: 100%


Steps to Reproduce:
1. Create/mount a gluster volume.
2. Copy a relatively large directory tree (i.e., kernel source tree).
3. ls -alR the resulting tree, observe RES/RSS of the glusterfs client process.
4. Replicate the directory tree and repeat step #3.
  
Actual results: With 4 replicated kernel source trees, I reproduce about 8GB of memory used in the client process.


Expected results: Some kind of cap on memory usage.


Additional info: It is expected that md-cache can result in significant memory usage compared to without the translator installed, but the user should be able to restrict the cache from growing beyond a certain point.

Comment 1 Brian Foster 2012-04-16 13:18:19 UTC

I'll try to add an lru mechanism to md-cache (i.e., similar to quick-read) to try and control this behavior...

Comment 2 Brian Foster 2012-04-17 14:31:31 UTC

Avati's comments:

"You make a good observation. The problem I suspect is because of the loaded quick-read translator which results in returning of file data in the lookup_cbk's xattr dict. The md-cache's xattr cache is a superset of which is needed (basically the entire xattr dictionary which also holds quick-read's response and not just the interested keys of md-cache). What is noted might very well be the same overrun behavior of quick-read, and for the same data element (rather than extended attributes). It would be interesting to see if we fix the *xatt_set() in md-cache to selectively dup just the required attributes and cache only those rather than blindly storing the entire dict as-is (and thereby caching file contents which were present to satisfy quick-read's request)."

---

I reproduce a reduction in glusterfs RSS from 28GB to 1.3GB with a particular dataset by removing quick-read from the client graph, which validates the above.

Comment 3 Brian Foster 2012-05-02 13:48:38 UTC

Suggestion from Amar:

< amarts> and about md-cache + quick-read consuming extra memory, we were thinking just doing a 'dict_del(GF_CONTENT_KEY);' in quick-read's lookup_cbk(), before unwind

Comment 4 Brian Foster 2012-05-03 15:06:06 UTC

http://review.gluster.com/3268

Comment 5 Anand Avati 2012-05-08 22:34:09 UTC

CHANGE: http://review.gluster.com/3268 (quick-read, md-cache: selectively cache xattr data to conserve memory) merged in master by Anand Avati (avati)

Comment 7 Sachidananda Urs 2012-12-18 06:50:23 UTC

Tested with more than 10 replicated kernel sources, metadata intensive operations. Memory usage more or less remained constant around 310M.

Couldn't even hit close to 1G. (Earlier hitting around 8G for a couple of replicated kernel sources).

Comment 8 Scott Haines 2013-09-23 22:32:40 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html