Bug 856459

Description of problem:
DHT-  lookup (stat, ls) on directory fails and gives error, 'invalid argument' if hashed sub-volume is down

Version-Release number of selected component (if applicable):


How reproducible:
not always

Steps to Reproduce:
1. Create a Distributed volume having 3 or more sub-volumes on multiple server and start that volume.

2. Fuse Mount the volume from the client-1 using “mount -t glusterfs  server:/<volume> <client-1_mount_point>”

3. From mount point create some dirs and files inside it

4. Find the hash value for dir name and also find its hashed sub volume

from test-client-1
[]# getfattr -n trusted.glusterfs.dht -e hex /home/test
getfattr: Removing leading '/' from absolute path names
# file: home/test
trusted.glusterfs.dht=0x00000001000000000000000055555554

hash value for file d10 is 4b423224 

so it should hash to sub-vol 'test-client-1'

5.bring hashed sub-vol down 

[root@Rhs3 test]# gluster volume status test
Status of volume: test
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick XX1:/home/test				24009	Y	11463
Brick XX2:/home/test				24009	N	16287
Brick XX3:/home/test				24009	Y	12116
NFS Server on localhost				38467	Y	16293
NFS Server on XXX				38467	Y	16005
NFS Server on xxx				38467	Y	15892

6. Execute ls or stat command from mount point

[root@a mnt]# ls
ls: cannot access d1: Invalid argument
ls: cannot access d3: Invalid argument
ls: cannot access d8: Invalid argument
ls: cannot access d10: Invalid argument
ls: cannot access d11: Invalid argument
ls: cannot access d18: Invalid argument
ls: cannot access d19: Invalid argument
ls: cannot access d20: Invalid argument
ls: cannot access d101: Invalid argument
ls: cannot access d105: Invalid argument
ls: cannot access d106: Invalid argument
d1   d101  d103  d105  d107  d11   d111  d113  d115  d117  d12  d14  d16  d18  d2   d3  d5  d7  d9
d10  d102  d104  d106  d109  d110  d112  d114  d116  d118  d13  d15  d17  d19  d20  d4  d6  d8  data1
[root@a mnt]# stat d1
stat: cannot stat `d1': Invalid argument

Actual results:
lookup results in invalid argument

Expected results:
though hashed sub-vol is down, if another sub-vol has files/diretories, it should list those files and directories rather than giving invalid argument.


Additional info:

this error will come whenever hashed sub-vol layout start-stop value is 0,0 as shown below.

[root@a mnt]# getfattr -m . -n trusted.glusterfs.pathinfo /home/racpatel/mnt
getfattr: Removing leading '/' from absolute path names
# file: home/racpatel/mnt
trusted.glusterfs.pathinfo="((<DISTRIBUTE:test-dht> <POSIX(/home/test):Rhs2:/home/test/>) (test-dht-layout (test-client-2 1431655765 2863311529) (test-client-0 2863311530 4294967295) (test-client-1 0 0)))"