Description of problem: DHT - after adding brick/s, layout for root directory is getting changed (without running rebalance) so files created on mount point after adding bricks are distributed to new brick/sub-volume also Version-Release number of selected component (if applicable): 3.3.0.7rhs-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: 1. Create a Distributed volume having 2 or more sub-volume and start the volume. [root@mia ~]# gluster volume info issue1 Volume Name: issue1 Type: Distribute Volume ID: c23ae123-742a-4420-b997-8c57ce94f4d7 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: fred.lab.eng.blr.redhat.com:/brick1/m1 Brick2: mia.lab.eng.blr.redhat.com:/brick1/m1 2. Fuse Mount the volume from the client-1 using “mount -t glusterfs server:/<volume> <client-1_mount_point>” mount -t glusterfs XXX:/issue1 /mnt/issue1 3. From mount point create some files at root level. for i in `seq 10 19` ; do dd if=/dev/zero of=/mnt/issue1/file_r${i} bs=16384 count=1 ; done 4.add brick/s in volume [root@mia ~]# gluster volume info issue1 Volume Name: issue1 Type: Distribute Volume ID: c23ae123-742a-4420-b997-8c57ce94f4d7 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: fred.lab.eng.blr.redhat.com:/brick1/m1 Brick2: mia.lab.eng.blr.redhat.com:/brick1/m1 Brick3: fan.lab.eng.blr.redhat.com:/brick1/m1 Brick4: cutlass.lab.eng.blr.redhat.com:/brick1/m1 5. create more files from mount point at root level for i in `seq 20 29` ; do dd if=/dev/zero of=/mnt/issue1/file_r${i} bs=16384 count=1 ; done 6. list all files on mountpoint 7. check on subvoulmes that files are distributed to new sub-volumes also. verify hash layout for new brick/sub-volume [root@cutlass issue1]# getfattr -d -m . -e hex /brick1/m1/ getfattr: Removing leading '/' from absolute path names # file: brick1/m1/ trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff trusted.glusterfs.volume-id=0xc23ae123742a4420b9978c57ce94f4d7 Actual results: After adding brick/s, hash-layout for root directory is getting changed (without running rebalance) Expected results: after adding brick/s,hash-layout for any existing dir should be changed when rebalance command is executed. Additional info:
Though the layouts do get change, this is not leading applications to see any errors. This change in Behaviour got introduced due to a fix for bug sent downstream: commit 8c4bb51e1f86c9726250832f05523d207c18a310 Author: shishir gowda <sgowda> Date: Tue Mar 12 14:51:14 2013 +0530 cluster/distribute: Ignore non-participating subvols for layout checks When subvols-per-directory is < available subvols, then there are layouts which are not populated. This leads to incorrect identification of holes or overlaps. We need to ignore layouts, which have err == 0, and start == stop. In the current scenario (start == stop == 0). For any other valid subvoles, err != 0 in case of layouts being zeroed out. Change-Id: I7e358b2829e4de249a77e0039960aec9055e1554 BUG: 884379 We've rectified the behaviour upstream and release-3.4 (Will update the bug when rhs-2.1 has the fix) with the fix: commit 9c1d7cebbc62723f719b2fd0c45e0a7452e0d6be Author: shishir gowda <sgowda> Date: Thu Apr 4 11:23:08 2013 +0530 cluster/distribute: Ignore non-participating subvols for layout checks When subvols-per-directory is < available subvols, then there are layouts which are not populated. This leads to incorrect identification of holes or overlaps. We need to ignore layouts, which have err == 0, and start == stop. In the current scenario (start == stop == 0). Additionally, in layout-merge, treat missing xattrs as err = 0. In case of missing layouts, anomalies will reset them. For any other valid subvoles, err != 0 in case of layouts being zeroed out. Also reverted back dht_selfheal_dir_xattr, which does layout calculation only on subvols which have errors. Change-Id: I9f57062722c9e8a26285e10675c31a78921115a1 BUG: 921408
Fix for bug 921408 is available as part of release glusterfs-3.4.0.8rhs-1. I was not able to reproduce the issue in the mentioned version. Please re-run the test and update the status as needed
verified on 3.4.0.8rhs-1.el6.x86_64 1. volume info [root@cutlass ~]# gluster v info bug Volume Name: bug Type: Distribute Volume ID: d96bcac3-5144-427b-80f7-b049af336c3c Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: fred.lab.eng.blr.redhat.com:/rhs/brick1/b1 Brick2: cutlass.lab.eng.blr.redhat.com:/rhs/brick1/b1 Brick3: fan.lab.eng.blr.redhat.com:/rhs/brick1/b1 Brick4: fan.lab.eng.blr.redhat.com:/rhs/brick1/b2 2. add-brick run rebalance and check status [root@fan ~]# gluster volume add-brick bug mia.lab.eng.blr.redhat.com:\/rhs/brick1/b1 volume add-brick: success [root@fan ~]# gluster volume rebalance bug start force volume rebalance: bug: success: Starting rebalance on volume bug has been successful. ID: bf0263a9-f73a-4888-a4e4-391205537d8e [root@fan ~]# gluster volume rebalance bug status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 20 0Bytes 111 0 completed 1.00 localhost 20 0Bytes 111 0 completed 1.00 localhost 20 0Bytes 111 0 completed 1.00 localhost 20 0Bytes 111 0 completed 1.00 mia.lab.eng.blr.redhat.com 0 0Bytes 165 0 completed 2.00 volume rebalance: bug: success: 3. again add-brick and from mount point create few files and check on backend [root@fan ~]# gluster volume add-brick bug mia.lab.eng.blr.redhat.com:/rhs/brick1/b2 volume add-brick: success mount:- [root@rhsauto037 bug]# for i in {1..20}; do touch fileo"$i" ; touch d1/fo"$i"; done mia:- [root@mia ~]# getfattr -d -m . -e hex /rhs/brick1/b2 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b2 trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 trusted.glusterfs.volume-id=0xd96bcac35144427b80f7b049af336c3c [root@mia ~]# ls /rhs/brick1/b2 d1 fileo10 fileo16 [root@mia ~]# ls /rhs/brick1/b2/d1 [root@mia ~]# getfattr -d -m . -e hex /rhs/brick1/b2/d1 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/b2/d1 trusted.gfid=0xb4217a8fe6984655b729e2bf900ae16f result:- after adding brick/s, layout for root directory is getting changed (without running rebalance) so files created on mount point after adding bricks are distributed to new brick/sub-volume also hence moving back to assigned
Fixed in version please.
verified with Big bend RHS ISO + glusterfs-*-3.4.0.35rhs-1.el6rhs.x86_64 need info :- -after adding brick it is not creating 'trusted.glusterfs.dht' attribute for newly added brick. So in a way, yes newly created files will not go there but Isn't it better to verify this after it creates attr for the newly added brick. Then we can say layout is getting changed or not? Already have bug for the attr is not self healed Bug 1005663 Let me know should I closed this and reopen again or should I verify this later.
Looks like the bug per description is fixed and tested. In such case, we should be marking this bug as VERIFIED and any further issue related to bug 1005663 should be filed as a new one.
verified with Big bend RHS ISO + glusterfs-*-3.4.0.35rhs-1.el6rhs.x86_64 Layout is not getting changed on adding brick so changing status to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html