Bug 1417028 - option performance.parallel-readdir can cause OOM in large volumes
Summary: option performance.parallel-readdir can cause OOM in large volumes
Alias: None
Product: GlusterFS
Classification: Community
Component: unclassified
Version: 3.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Poornima G
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-27 02:55 UTC by Raghavendra G
Modified: 2017-03-06 17:44 UTC (History)
1 user (show)

Fixed In Version: glusterfs-3.10.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-03-06 17:44:28 UTC
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:

Attachments (Terms of Use)

Description Raghavendra G 2017-01-27 02:55:55 UTC
Description of problem: With performance.parallel-readdir enabled, a readdir-ahead is loaded in each subvolume of DHT. Though individual readdir-ahead has a limit on the amount of memory used for caching, there is no limit on the number of subvolumes to DHT and hence the number of readdir-ahead in graph can be large. The cumulative cache size of all readdir-ahead xlators can be huge in such a scenario. So, there should be a way to limit the amount of total cache consumed by all readdir-ahead xlators in a large volume

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 1 Worker Ant 2017-01-27 02:58:19 UTC
REVIEW: https://review.gluster.org/16459 (glusterd, rda: If parallel readdir is enabled, split the cache limit) posted (#2) for review on release-3.10 by Raghavendra G (rgowdapp@redhat.com)

Comment 2 Worker Ant 2017-01-27 14:03:03 UTC
COMMIT: https://review.gluster.org/16459 committed in release-3.10 by Shyamsundar Ranganathan (srangana@redhat.com) 
commit 6ebbfb67315ae9abc4058775d3b48d5abfe306d5
Author: Poornima G <pgurusid@redhat.com>
Date:   Tue Jan 17 17:45:59 2017 +0530

    glusterd, rda: If parallel readdir is enabled, split the cache limit
    With patch http://review.gluster.org/#/c/16072/ readdir-ahead can be
    loaded as a child of dht. i.e. there can be more than one instance
    of readdir-ahead in client process. In this case the rda-cache-size
    should be split among all the readdir-ahead instances. Also the
    value of rda-request-size is considered as the minimum cache size
    of any readdir-ahead instance.
    >Change-Id: Iea2fe6d4c46adc09dd2e9a252332a0fe3005f2b9
    >BUG: 1401812
    >Signed-off-by: Poornima G <pgurusid@redhat.com>
    >Reviewed-on: https://review.gluster.org/16424
    >Smoke: Gluster Build System <jenkins@build.gluster.org>
    >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    >CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    >Reviewed-by: Kaushal M <kaushal@redhat.com>
    >Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    Change-Id: Iea2fe6d4c46adc09dd2e9a252332a0fe3005f2b9
    BUG: 1417028
    Signed-off-by: Poornima G <pgurusid@redhat.com>
    (cherry picked from commit f245dc568e3c22882e22ddd3e26a4207f5704e3b)
    Reviewed-on: https://review.gluster.org/16459
    Tested-by: Raghavendra G <rgowdapp@redhat.com>
    Smoke: Gluster Build System <jenkins@build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.org>
    Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>

Comment 3 Shyamsundar 2017-03-06 17:44:28 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report.

glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html
[2] https://www.gluster.org/pipermail/gluster-users/

Note You need to log in before you can comment on or make changes to this bug.