> Based on the statedump, one big leak I see is from dirents which are allocated by DHT but don't seem to be leaking in dht. I think some xlator above dht is not freeing it. Could you let me know the size of the directory you may have?
I think this can be readdir-ahead. Is it possible to turn off readdir-ahead and see whether it helps?
(In reply to Raghavendra G from comment #2)
> > Based on the statedump, one big leak I see is from dirents which are allocated by DHT but don't seem to be leaking in dht. I think some xlator above dht is not freeing it. Could you let me know the size of the directory you may have?
> I think this can be readdir-ahead. Is it possible to turn off readdir-ahead
> and see whether it helps?
The reason I suspect readdir-ahead is that there is no upper limit to amount of dentries readdir-ahead can store as of now. It keeps populating the cache till EOD is reached or an error is encountered in readdir from lower xlators. So, in a scenario where readdirs from application are infrequent and directory is huge, all the dentries of a directory is cached in memory and that could result in OOM. Please note that it is not a leak, but a bug in readdir-ahead to not have an upper limit.
I have performed the same steps with "performance.readdir-ahead off" with gluster 126.96.36.199 build and i am not hitting this issue
As per the triaging we all have the agreement that this BZ has to be fixed in rhgs-3.2.0. Providing qa_ack
The unlimited caching behaviour of readdir-ahead is been thee from day-0. Implementing the upper cache limit for readdir-ahead in 3.2 time frame is difficult as the fix is intrusive.
Can this be deferred for 3.2?
An upstream patch http://review.gluster.org/#/c/16137/ posted for review
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/93587
a compilation failure was introduced by https://code.engineering.redhat.com/gerrit/#/c/93587 which is now fixed through https://code.engineering.redhat.com/gerrit/#/c/93622/ (this issue was only there in downstream code)
We'd need to pull in one more patch here to ensure rda-low-wmark & rda-high-wmark options are not exposed to the users.
upstream mainline patch : http://review.gluster.org/16297
downstream patch : https://code.engineering.redhat.com/gerrit/#/c/93820/
BZ added to erratum https://errata.devel.redhat.com/advisory/24866
Moving to ON_QA
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.