Bug 860663

Summary:	DHT - Hash layout for Directory is changing when sub-volume is down and user does a rebalance operation, as a result at same directory level duplicate files can be created
Product:	[Community] GlusterFS	Reporter:	shishir gowda <sgowda>
Component:	distribute	Assignee:	shishir gowda <sgowda>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	amarts, fharshav, gluster-bugs, kirantpatil, nsathyan, racpatel, rhs-bugs, sgowda, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	860599	Environment:
Last Closed:	2013-07-24 17:41:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	860599
Bug Blocks:

Description shishir gowda 2012-09-26 12:30:51 UTC

+++ This bug was initially created as a clone of Bug #860599 +++

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

--- Additional comment from racpatel on 2012-09-26 05:42:53 EDT ---

Created attachment 617484 [details]
log

--- Additional comment from racpatel on 2012-09-26 05:43:35 EDT ---

Created attachment 617485 [details]
rebalance log

--- Additional comment from racpatel on 2012-09-26 05:50:04 EDT ---

Description of problem:
DHT - Hash layout for Directory is getting changed when sub-volume is down and user performs rebalance operation, as a result at same level multiple files can be created having same name

Version-Release number of selected component (if applicable):
3.3.0rhs-28.el6rhs.x86_64

How reproducible:
always

Steps to Reproduce:

1. Create a Distributed volume having 3 or more sub-volumes on multiple server and start that volume.

2. Fuse Mount the volume from the client-1 using “mount -t glusterfs  server:/<volume> <client-1_mount_point>”

3. From mount point create some dirs and files inside it. verify has layout for directory/root

[root@Rhs1 t1]# getfattr -d -m . -e hex /home/t1
getfattr: Removing leading '/' from absolute path names
# file: home/t1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff
trusted.glusterfs.volume-id=0x0e78545678f542e084fab375a89b813c

[root@Rhs2 t1]# getfattr -d -m . -e hex /home/t1
getfattr: Removing leading '/' from absolute path names
# file: home/t1
security.selinux=0x73797374656d5f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000000000000055555554
trusted.glusterfs.volume-id=0x0e78545678f542e084fab375a89b813c

[root@Rhs3 t1]# getfattr -d -m . -e hex /home/t1
getfattr: Removing leading '/' from absolute path names
# file: home/t1
security.selinux=0x73797374656d5f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9
trusted.glusterfs.volume-id=0x0e78545678f542e084fab375a89b813c



4. Bring on of the sub-volume down.

5. Execute rebalance.
gluster volume rebalance <volname> start

6. check the hash layout for directory/root level. It is changed.


[root@Rhs1 t1]# getfattr -d -m . -e hex /home/t1
getfattr: Removing leading '/' from absolute path names
# file: home/t1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0x0e78545678f542e084fab375a89b813c

[root@Rhs2 t1]# getfattr -d -m . -e hex /home/t1
getfattr: Removing leading '/' from absolute path names
# file: home/t1
security.selinux=0x73797374656d5f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.volume-id=0x0e78545678f542e084fab375a89b813c

7. Now from mount point try to create files which exixts in down sub-vol . Files would be created successfully
8. Bring all the sub-vols up
9. Execute ls command from mount point , multiple files with same name will be present.

[root@client test]# ls
f1   f11  f13  f15  f17  f19  f20  f22  f24  f26  f28  f3   f31  f33  f35  f37  f39  f40  f6  f8
f10  f12  f14  f16  f18  f2   f21  f23  f25  f27  f29  f30  f32  f34  f36  f38  f4   f5   f7  f9
[root@client test]# touch f11
[root@client test]# touch f14
[root@client test]# ls
f1   f11  f12  f14  f15  f17  f19  f20  f22  f24  f26  f28  f3   f31  f33  f35  f37  f39  f40  f6  f8
f10  f11  f13  f14  f16  f18  f2   f21  f23  f25  f27  f29  f30  f32  f34  f36  f38  f4   f5   f7  f9

  
Actual results:
rebalance is fixing directory lay out eventhough one brick/sub-vol is down

Expected results:
rebalance should not fix directory layout when any brick/sub-vol is down

Comment 1 shishir gowda 2012-10-11 06:03:16 UTC

Fix sent for review (http://review.gluster.org/#change,4000)

To test fix:

1. Brick down issue
   a. Follow steps provided above.
2. Brick out of space issue 
   a. Create a brick with low available space, and create a >2 bricks dht volume
   b. Fill the bricks with i/o on any files on the brick
   c. add-brick and rebalance
   d. New layouts should be written.

Comment 2 Vijay Bellur 2012-11-30 06:11:18 UTC

CHANGE: http://review.gluster.org/4000 (cluster/dht: fail fix-layout if any of the subvol is down) merged in master by Anand Avati (avati)

Comment 3 Vijay Bellur 2012-12-03 05:40:46 UTC

CHANGE: http://review.gluster.org/4254 (tests: Test the exit status of touch) merged in master by Vijay Bellur (vbellur)

Comment 4 Kiran 2014-08-27 09:49:34 UTC

Version-Release number of selected component (if applicable):
gluster v3.5.2

How reproducible:
Always

Steps to Reproduce:
1. Install gluster v3.5.2 rpm on CentOS 6.4
2. clone the gluster from github and checkout v3.5.2 
3. DEBUG=1 prove tests/bugs/bug-860663.t

Actual results:
System hangs

Expected results:
Testcase should pass

Additional info:
[root@fractal-0e6e glusterfs]# uname -a
Linux fractal-0e6e.fractal.lan 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux