Description of problem:
File creation fails intermittently on a distributed volume, while trying to write concurrently from two clients at the same time. 


Version-Release number of selected component (if applicable):

glusterfs-3.7.1-11.el6rhs.x86_64
glusterfs-server-3.7.1-11.el6rhs.x86_64

Red Hat Gluster Storage Server 3.1

How reproducible:

Not always , Intermittent 


Steps to Reproduce:

1. Create a distributed volume having 60 bricks or a relative high number of bricks on different nodes.

2. Mount the volume on two different clients. 

3. Try to create directories/files from simultaneously from both the clients using the below script. 

for i in {1..9}
do
mkdir /mnt/test/$i
cd /mnt/test/$i
touch $HOSTNAME.$i
done

Actual results:

For some files , it fails to create from one of the server. At times it gives , the "No such file/directory" for the file , like below. 

[root@server2 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/1': File exists
mkdir: cannot create directory `/mnt/test/2': File exists
mkdir: cannot create directory `/mnt/test/5': File exists
mkdir: cannot create directory `/mnt/test/6': File exists
touch: cannot touch `server2.6': No such file or directory
mkdir: cannot create directory `/mnt/test/7': File exists
mkdir: cannot create directory `/mnt/test/9': File exists

[root@server1 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/3': File exists
mkdir: cannot create directory `/mnt/test/4': File exists
mkdir: cannot create directory `/mnt/test/8': File exists


And at some different point of time it failed giving , stale file handle. 

[root@server2 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/4': File exists
touch: cannot touch `server2.4': Stale file handle
mkdir: cannot create directory `/mnt/test/6': File exists
mkdir: cannot create directory `/mnt/test/7': File exists
mkdir: cannot create directory `/mnt/test/8': File exists
mkdir: cannot create directory `/mnt/test/9': File exists



[root@server1 ~]# ./test.sh
mkdir: cannot create directory `/mnt/test/1': File exists
mkdir: cannot create directory `/mnt/test/2': File exists
mkdir: cannot create directory `/mnt/test/3': File exists
mkdir: cannot create directory `/mnt/test/5': File exists


Expected results:

The file creation should work without any issue. 


Additional info:

[2016-01-16 02:19:01.799335] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2016-01-16 03:00:36.570750] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 936606817: MKDIR() /pathtofile/0006/897 => -1 (File exists)
[2016-01-16 03:00:36.667744] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 29-volname-dht: Found anomalies in /pathtofile/0006/897 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2016-01-16 03:00:36.679985] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-62: remote operation failed [Stale file handle]
[2016-01-16 03:00:36.679982] W [MSGID: 114031] [client-rpc-fops.c:2322:client3_3_setattr_cbk] 29-volname-client-63: remote operation failed [Stale file handle]