Bug 1202675

+++ This bug was initially created as a clone of Bug #1202673 +++

+++ This bug was initially created as a clone of Bug #1202669 +++

+++ This bug was initially created as a clone of Bug #1202541 +++

Description of problem:

ls -l performance has been greatly reduced on replicated volumes.  I do not see this perf hit on distributed volumes.

Version-Release number of selected component (if applicable):


How reproducible:

Every time.

Steps to Reproduce:
1.  Create 10k files
2.  Clear buffers and cache on both servers and clients
3.  Run ls -l on files

Actual results:



Expected results:


Additional info:

--- Additional comment from Ben Turner on 2015-03-16 18:51:55 EDT ---
The commit that causes perf degrade has been found to be the following:

<commit details>
Author: Krutika Dhananjay <kdhananj>
Date:   Thu Jan 22 17:02:20 2015 +0530

     cluster/afr: When parent and entry read subvols are different, set 
entry->inode to NULL
</commit details>


--- Additional comment from Krutika Dhananjay on 2015-03-17 03:19:56 EDT ---

Let me briefly explain what the problem with AFR was:
In AFR, every file has a "read child" associated with it. Read operations on a file (like readv under the data read category, getxattr, stat etc under metadata read category and readdirp under entry read category) are always
served from the designated read child of the file/dir unless it contains the bad copy of the file (i.e., in need of a self-heal). I can think of atleast two reasons why this is useful:
a. it is sufficient to serve reads from only one of the copies of a file, since all copies are identical under normal circumstances.
b. Certain attributes like mtime/ctime/atime might differ across different copies of a file on different bricks due to clock skew across the servers. In these cases, it is good to always return the same values across consecutive requests for these attributes, because
    we do not want the application to mistakenly think that the file underwent some change just because AFR returned different values of timestamps across different calls.

The problem case is readdirp. Readdirp also fetches the attributes of the entries read. A directory on which readdirp is performed can have a read child x while some of the entries read could have read child y.
This means that readdirp may violate (b) above in terms of giving timestamps from a copy of the entry which is not from its read child. The other problem with this behavior is that, even if the directory's read child
had a bad copy of some of the entries, this will not be detected. My patches fixed these issues with readdirp by forcing a lookup on those read entries whose read child did not match that of the parent. This would have led to some extra lookups.

Both the BZs above were manifestations of the same bug in readdirp just described. 



[UPDATE]
It was decided in the meeting that the behavior introduced by the AFR patch is to be made optional. And by default, the behavior would be "off".