Description of problem:
DHT :- For file if cached sub-volume is down and hashed sub-volume is up then  many operations which acees/modify file data/meta data(cp,chmod,mv,cat etc)  removes link file(hashed file) from hashed sub-volume


Version-Release number of selected component (if applicable):
3.3.0.5rhs-40.el6rhs.x86_64


How reproducible:
always

Steps to Reproduce:
1. Create a Distributed volume having 3 or more sub-volumes on multiple server and start that volume.

2. Fuse Mount the volume from the client-1 using “mount -t glusterfs  server:/<volume> <client-1_mount_point>”

3. From mount point create some dirs and files inside it.

4. rename files.

5. bring one sub-volume down by killing brick process

6.Now choose files for which hashed sub volume is up and cached sub-volume is down . 
- Execute mv,cp,cat commands on those files and check bricks.


Before those operation, link file is present on brick. 
Any attempt to access file(cached sub-volume down and hashed sub-volume up) from mount point give error no such file but after that link file is removed from bricks

e.g.
**cat** 

(up) server 1:- 
[root@dhcp159-157 bdown1]# ls -l renamefile11 
---------T 2 root root 0 Jan 24 13:16 renamefile11 

(down) server 3 :-
[root@dhcp159-165 bdown1]# ls -l renamefile11 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile11 

mount 
[root@dhcp159-207 bdown1]# cat renamefile11 
cat: renamefile11: No such file or directory 

(up) server 1:- 
[root@dhcp159-157 bdown1]# ls -l renamefile11 
ls: cannot access renamefile11: No such file or directory 

(down) server 3 :-
[root@dhcp159-165 bdown1]# ls -l renamefile11 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile11 


**modify xattr** 

(up) server 1:- 
[root@dhcp159-157 bdown1]# ls -l renamefile15 
---------T 2 root root 0 Jan 24 13:16 renamefile15 

(down) server 3 :-
[root@dhcp159-165 bdown1]# ls -l renamefile15 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile15 

mount 
[root@dhcp159-207 bdown1]# setfattr -n user.abc -v kkk renamefile15 
setfattr: renamefile15: No such file or directory 

(up) server 1:- 
[root@dhcp159-157 bdown1]# ls -l renamefile15 
ls: cannot access renamefile15: No such file or directory 

(down) server 3 :-
[root@dhcp159-165 bdown1]# ls -l renamefile15 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile15 


** chmod ** 

(up) server 2:- 
[root@dhcp159-158 bdown1]# ls -l renamefile25 
---------T 2 root root 0 Jan 24 13:16 renamefile25 


(down) server 3:- 
[root@dhcp159-165 bdown1]# ls -l renamefile25 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile25 

mount 
[root@dhcp159-207 bdown1]# chmod 777 renamefile25 
chmod: cannot access `renamefile25': No such file or directory 

(up) server 2:- 
[root@dhcp159-158 bdown1]# ls -l renamefile25 
ls: cannot access renamefile25: No such file or directory 

(down) server 3:- 
[root@dhcp159-165 bdown1]# ls -l renamefile25 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile25 


***cp *** 
(up) server 1:- 
[root@dhcp159-157 bdown1]# ls -l renamefile26 
---------T 2 root root 0 Jan 24 13:34 renamefile26 


(down) server 3:- 
[root@dhcp159-165 bdown1]# ls -l renamefile26 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile26 

mount 
[root@dhcp159-207 bdown1]# cp renamefile26 renamefile101 
cp: cannot stat `renamefile26': No such file or directory 

(up) server 1 
[root@dhcp159-157 bdown1]# ls -l renamefile26 
ls: cannot access renamefile26: No such file or directory 

(down) server 3 
[root@dhcp159-165 bdown1]# ls -l renamefile26 
-rw-r--r-- 2 root root 0 Jan 24 13:16 renamefile26
  

Actual results:
if cached sub-volume is down and hashed sub-volume is up then  many operation that access file data or modify file meta data(cp,chmod,mv,cat etc)  removes link file(hashed file) from hashed sub-volume

Expected results:
link file should not be deleted from bricks, if user tries to aceess/modify file data/metadat.

Additional info: