Bug 1118574

+++ This bug was initially created as a clone of Bug #1116376 +++

Description of problem:
-------------------------
mkdir on few directories failed on a distribute-replicate volume when bricks were added to the volume. 

root@dj [Jul-04-2014-15:41:44] >mkdir -p A{101..200}/B{1..5}/C{1..10}
mkdir: cannot create directory `A105/B5/C6': Stale file handle
mkdir: cannot create directory `A105/B5/C9': Stale file handle

root@dj [Jul-04-2014-16:49:18] >ls -l A105/B5/C6
ls: cannot access A105/B5/C6: No such file or directory

root@dj [Jul-04-2014-16:49:21] >ls -l A105/B5/C9
ls: cannot access A105/B5/C9: No such file or directory
root@dj [Jul-04-2014-16:49:24] >

Version-Release number of selected component (if applicable):
=================================================================
glusterfs 3.6.0.22 built on Jun 23 2014 10:33:07

How reproducible:
=================
First time saw the error and reporting it. 

Steps to Reproduce:
====================
1. Create a dis-rep volume ( 2 x 2 ) . Start the volume. 

2. Create fuse/nfs mount. start creating files/dirs on all the mounts. 

3. From one fuse mount execute : "mkdir -p A{101..200}/B{1..5}/C{1..10}". (No other mounts were create any files/directories on the above newly created dirs. They were doing io's on other dirs/files)

4. While mkdir is in process, add-bricks to the volume : 


Actual results:
=================
mkdir failed with "Stale File Handle"

root@dj [Jul-04-2014-15:41:44] >mkdir -p A{101..200}/B{1..5}/C{1..10}
mkdir: cannot create directory `A105/B5/C6': Stale file handle
mkdir: cannot create directory `A105/B5/C9': Stale file handle


Expected results:
====================
mkdir shouldn't fail.


comment from Raghavendra G:
---------------------------
Till [1], the layouts of all the directories created after an add-brick, included the newly added brick. However, this behaviour had also resulted in a bug [2]. Consider the following set of operations:

1. cd /mnt/glusterfs
2. mkdirp -p 1/2/3/4/5/6
3. cd 1/2/3/4/5/6
4. add a new brick
5. mkdir -p 7/<a-brick-name-which-hashes-to-newly-added-brick>

Here mkdir of 7/<a-brick-name-which-hashes-to-newly-added-brick> fails with ENOENT. This is because on the newly added-brick there is no guarantee that /mnt/glusterfs/1/2/3/4/5/6/7 is created - mkdir (7) on newly-added-brick could've returned ENOENT because of its parents not being present - and layout of directory 7 used to include the newly added brick. Since mkdir (<a-brick-name-which-hashes-to-newly-added-brick>) fails on hashed-subvol, the directory creation is aborted and an ENOENT is returned to application. This is the RCA of [2].

Now, with [1], the errno returned as part of failure of mkdir (7) on newly-added-brick is ESTALE and dht doesn't consider the newly-added-brick as part of the layout of directory 7 and hence no failures are seen while creating children of 7 and [2] is fixed. However [3] seems to have nullified effect of [1] even in newer client. If I revert [3], the issue seems fixed. Also this bug is seen only in release-3.5 branches where [3] is merged, but not on master where [3] is not present.

[1] http://review.gluster.org/6318
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1006809
[3] http://review.gluster.org/#/c/8080/