It turns out readdir chunking, as currently implemented in Ganesha, cannot function correctly on Gluster.  It depends on stable d_off from readdir() across directory streams (opendir() -> closedir()), which is not something that Gluster can provide at this time.  Since this tends to work, especially for small or simple directory structures, this was not discovered until extensive testing was done.

This can be fixed, but not in a short timeframe.  It would have to delay the 3.4 release.

It is not correct to say it was validated by QE; to me that implies that it worked.

QE tested the feature and found it did not work correctly.

Also in discussions with Poornima we have learned that some of the assumptions that were made about glusterfs — and gfapi in particular — are not valid.

Dan has a partial fix which we will evaluate for potential inclusion in 3.4.1.

Just for my understanding, re-iterating few points -

Readdir chunking feature mainly addresses below issue -
(1) Earlier when one client does readdir, other client's fops involving lookup 
on any of those dirents get blocked till the entire traversal of directory is done (which was taking several hours to finish for larger workload)

(2) With readdir chunking, we read chunk by chunk and lock/unlock directory locks for every chunk so that other clients' operations continue in between.

(3) Now dirents are cached as part of dirent cache but each looked up entry with attributes is cached in md-cache.

As part of readdir chunk, while reading from a particular offset, if the entry is found in dirent cache but not in md-cache, we try to reload entire chunk (if fsal_readdir_plus option is enabled). Otherwise, we need to perform lookup for all the subsequent dirents in that chunk which could affect performance. 

This chunk reloading is what causing issue (- i.e, getting duplicate /missing entries) but not the readdir chunk feature to begin with. 

(4) Now with https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/425282, if we can disable fsal_readdir_plus for gluster, but keep readdir chunk feature enabled, only chunk reloading doesnt happen but issue#1 is still addressed as we still read dirents chunk by chunk. 

However this change may affect listing performance but should n't be worse when compared to disabling readdir chunk all together.

(5) Also, till now, dirent cache limit was to set to higher value when compared to md-cache limit. If we keep these 2 values same, do we still run into case#3?

Request Dan/Frank to provide comments and confirm if my understanding above is correct. Thanks!

1.  Correct.  It also blocked the readdir in progress, not just other clients.

2. Correct.

3. The reloading-a-chunk feature enabled by fsal_readdir_plus is not the problem, it just reliably shows up the problem.  I can hit the problem with a straight chunked readdir without that feature under certain circumstances.  

4.  No, because just chunking is enough.

5.  Yes.

In addition to disabling fsal_readdir_plus, we also need to keep the dir FD open for the entire duration of the readdir.  I'm working on a patch for that.  This will get us part of the features of chunking:  The part where the initial load is broken up.  It won't, however, get us the ability to load a subset of the directory at need; if any chunk of the directory is recycled, then we need to toss the entire directory and reload it next time.  This will have a noticeable affect on large workloads.

In order to have full working chunking, Gluster needs to guarantee the order of dirents and offset of dirents are stable across readdirs (assuming that the directory is not changing, of course).

Thank you very much for the clarification Dan. Have one basic doubt though ..

(In reply to Daniel Gryniewicz from comment #11)

> 3. The reloading-a-chunk feature enabled by fsal_readdir_plus is not the
> problem, it just reliably shows up the problem.  I can hit the problem with
> a straight chunked readdir without that feature under certain circumstances.
> 
> 
> 4.  No, because just chunking is enough.
> 
> 5.  Yes.
> 
> In addition to disabling fsal_readdir_plus, we also need to keep the dir FD
> open for the entire duration of the readdir.  I'm working on a patch for
> that.  This will get us part of the features of chunking:  The part where
> the initial load is broken up.  It won't, however, get us the ability to
> load a subset of the directory at need; if any chunk of the directory is
> recycled, then we need to toss the entire directory and reload it next time.
> This will have a noticeable affect on large workloads.
> 
> In order to have full working chunking, Gluster needs to guarantee the order
> of dirents and offset of dirents are stable across readdirs (assuming that
> the directory is not changing, of course).

In that case, even without readdir chunk, we have new glfd open for each call of read_dirents() and are reading from certain offset. Then why is this issue not seen then or is it just that the chance of hitting it is very low without chunking?

I wouldn't be surprised if it does fail without chunking.  I haven't tested that, because that code is gone from 2.6 and later.

My guess (and this is only a guess) is that it works because the time is very short, since MDCACHE loads the entire directory without any round-trips to the client.  This may be completely wrong, though.

(In reply to Daniel Gryniewicz from comment #13)
> I wouldn't be surprised if it does fail without chunking.  I haven't tested
> that, because that code is gone from 2.6 and later.
> 
> My guess (and this is only a guess) is that it works because the time is
> very short, since MDCACHE loads the entire directory without any round-trips
> to the client.  This may be completely wrong, though.

Okay .. so even in the current code (>=2.6), with readdir chunk disabled (Dir_Chunk=0), there are chances (though remote) that we could still get duplicate/missing entries ..right?

Until my WIP lands, yes.  Probably even in 2.5.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3252