Description of problem:
=======================
DHT :- unable to access data of file from mount after rebalance and multiple rename


Version-Release number of selected component (if applicable):
=============================================================
3.6.0.28-1.el6rhs.x86_64



How reproducible:
=================
intermittent

Steps to Reproduce:
===================
1. created Distributed volume (4 brick) , start it and mount it.
2. create 100 files on mount f{1..100}-1
3. add  more bricks
4. start rebalance
5. start renaming those file from nfs mount in parellal
keep sending lookup from another mount.
6. When rebalance is over unable to acces data from one file

tried to access data from nfs as well as FUSE mount

FUSE mount :-

[root@dht17 screw]# cp f60-8 mvdir1
cp: cannot open `f60-8' for reading: No such file or directory

[root@dht17 screw]# stat f60-8
  File: `f60-8'
  Size: 0               Blocks: 0          IO Block: 131072 regular empty file
Device: 27h/39d Inode: 12830275877193656721  Links: 0
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2014-09-09 15:14:39.000000000 -0400
Modify: 2014-09-09 15:14:39.000000000 -0400
Change: 2014-09-09 15:15:50.160000805 -0400

[root@dht17 screw]# cat f60-8 
cat: f60-8: No such file or directory

Actual results:
===============
file is listed but unable to access/modify its data 


Expected results:
=================
All files should be accessible from mount


Additional info:
================
check the log, that file is treated as directory
[root@dht17 screw]# tail -f /var/log/glusterfs/mnt-screw.log 
[2014-09-10 17:21:26.302580] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 2-screw-client-1: remote operation failed: No such file or directory. Path: /f60-8 (bcb7bdec-9123-483c-b20e-4beebe642991)
[2014-09-10 17:21:26.303437] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 2-screw-client-3: remote operation failed: No such file or directory. Path: /f60-8 (bcb7bdec-9123-483c-b20e-4beebe642991)
[2014-09-10 17:21:26.304488] I [dht-layout.c:663:dht_layout_normalize] 2-screw-dht: Found anomalies in /f60-8 (gfid = bcb7bdec-9123-483c-b20e-4beebe642991). Holes=1 overlaps=0
[2014-09-10 17:21:26.304563] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: bcb7bdec-9123-483c-b20e-4beebe642991: failed to resolve (Stale file handle)
[2014-09-10 17:21:26.304592] E [fuse-bridge.c:2064:fuse_open_resume] 0-glusterfs-fuse: 1946345: OPEN bcb7bdec-9123-483c-b20e-4beebe642991 resolution failed
[2014-09-10 17:24:05.196679] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 2-screw-client-1: remote operation failed: No such file or directory. Path: /f60-8 (bcb7bdec-9123-483c-b20e-4beebe642991)
[2014-09-10 17:24:05.196985] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 2-screw-client-3: remote operation failed: No such file or directory. Path: /f60-8 (bcb7bdec-9123-483c-b20e-4beebe642991)
[2014-09-10 17:24:05.198512] I [dht-layout.c:663:dht_layout_normalize] 2-screw-dht: Found anomalies in /f60-8 (gfid = bcb7bdec-9123-483c-b20e-4beebe642991). Holes=1 overlaps=0
[2014-09-10 17:24:05.198629] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: bcb7bdec-9123-483c-b20e-4beebe642991: failed to resolve (Stale file handle)
[2014-09-10 17:24:05.198679] E [fuse-bridge.c:2064:fuse_open_resume] 0-glusterfs-fuse: 1946363: OPEN bcb7bdec-9123-483c-b20e-4beebe642991 resolution failed



#0  dht_layout_normalize (this=0x11aad40, loc=0x7f8dac2f1920,
layout=0x7f8da0000ab0) at dht-layout.c:625
#1  0x00007f8daf3d6a3f in dht_discover_complete (this=0x11aad40,
discover_frame=0x7f8dbcd7d8e8) at dht-common.c:202
#2  0x00007f8daf3de493 in dht_discover_cbk (frame=0x7f8dbcd7d8e8,
cookie=0x7f8dbcf46b0c, this=0x11aad40, op_ret=<value optimized out>,
op_errno=116, inode=0x7f8da7b21d64, stbuf=0x7ffff8ca3f00, xattr=0x0,
postparent=0x7ffff8ca3e90)
     at dht-common.c:366
#3  0x00007f8db4052267 in client3_3_lookup_cbk (req=<value optimized
out>, iov=<value optimized out>, count=<value optimized out>,
myframe=0x7f8dbcf46b0c) at client-rpc-fops.c:2767
#4  0x00000033ba60e9c5 in rpc_clnt_handle_reply (clnt=0x119af30,
pollin=0x15d0270) at rpc-clnt.c:763
#5  0x00000033ba60fe4f in rpc_clnt_notify (trans=<value optimized out>,
mydata=0x119af60, event=<value optimized out>, data=<value optimized
out>) at rpc-clnt.c:891
#6  0x00000033ba60b668 in rpc_transport_notify (this=<value optimized
out>, event=<value optimized out>, data=<value optimized out>) at
rpc-transport.c:512
#7  0x00007f8db4693275 in socket_event_poll_in (this=0x11fc880) at
socket.c:2127
#8  0x00007f8db4694c5d in socket_event_handler (fd=<value optimized
out>, idx=<value optimized out>, data=0x11fc880, poll_in=1, poll_out=0,
poll_err=0) at socket.c:2240
#9  0x00000033b9e76367 in event_dispatch_epoll_handler
(event_pool=0x1016d00) at event-epoll.c:384
#10 event_dispatch_epoll (event_pool=0x1016d00) at event-epoll.c:445
#11 0x0000000000407e93 in main (argc=4, argv=0x7ffff8ca5398) at
glusterfsd.c:2023