REVIEW: http://review.gluster.org/8577 (CLI: Adding warning message in case remove-brick commit executed) posted (#1) for review on master by susant palai (spalai)
Is this issue applicable to 3.6 ,3.5 and 3.4 release branch?
For those who can't see the private version (mostly internal process stuff) here are the good bits. Description of problem: DHT - remove-brick - data loss - when remove-brick with 'start' is in progress, perform rename operation on files. commit remove-brick operation, after status is 'completed' and few files are missing. (not related to defect - 963896 - hash layout is not problem here) Version-Release number of selected component (if applicable): 3.3.0.10rhs-1.el6.x86_64 How reproducible: always Steps to Reproduce: 1.had a cluster of 3 RHS server and DHT volume having four bricks mounted as fuse [root@rhsauto031 ~]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 0130dae0-0573-491b-a4b2-14ac872624e7 Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick2 Brick2: rhsauto038.lab.eng.blr.redhat.com:/rhs/brick2 Brick3: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick2 Brick4: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 [root@localhost rtest]# mount | grep test glusterfs#rhsauto018.lab.eng.blr.redhat.com:/dist on /mnt/rtest type fuse (rw,default_permissions,allow_other,max_read=131072) 2. run remove brick with start option [root@rhsauto038 rpm]# gluster volume remove-brick dist rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 start Remove Brick start successful [root@rhsauto038 rpm]# gluster volume remove-brick dist rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0 0 0 not started rhsauto031.lab.eng.blr.redhat.com 0 0 0 0 not started rhsauto018.lab.eng.blr.redhat.com 668 29360128 2054 0 in progress 3. while data migration is in progress rename files from mount point mount point :- [root@localhost rtest]# for i in {1..100}; do mv d1/f$i d1/filenew$i; done [root@localhost rtest]#for i in {1..100}; do for j in {1..100}; do mv d$j/f$i d$j/filenew$i;done; done [root@localhost rtest]# ls d25/filenew25 d25/filenew25 4. verify that data - migration is in progress or not and check hash range for removed brick [root@rhsauto018 rpm]# getfattr -d -m . -e hex /rhs/brick10/ getfattr: Removing leading '/' from absolute path names # file: rhs/brick10/ trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000000000000000000000 trusted.glusterfs.volume-id=0x0130dae00573491ba4b214ac872624e7 [root@rhsauto018 rpm]# getfattr -d -m . -e hex /rhs/brick10/d25 getfattr: Removing leading '/' from absolute path names # file: rhs/brick10/d25 trusted.gfid=0xb5cc8353864e4a2dbdee8afabd099686 trusted.glusterfs.dht=0x00000001000000000000000000000000 [root@rhsauto038 rpm]# gluster volume remove-brick dist rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0 0 0 not started rhsauto031.lab.eng.blr.redhat.com 0 0 0 0 not started rhsauto018.lab.eng.blr.redhat.com 6839 1062207488 21350 0 in progress 5. Once migration is completed, perform commit for remove-brick [root@rhsauto038 rpm]# gluster volume remove-brick dist rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0 0 0 not started rhsauto031.lab.eng.blr.redhat.com 0 0 0 0 not started rhsauto018.lab.eng.blr.redhat.com 7165 1138753536 22257 0 completed [root@rhsauto038 rpm]# gluster volume remove-brick dist rhsauto018.lab.eng.blr.redhat.com:/rhs/brick10 commit Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y Remove Brick commit successful [root@rhsauto038 rpm]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 0130dae0-0573-491b-a4b2-14ac872624e7 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick2 Brick2: rhsauto038.lab.eng.blr.redhat.com:/rhs/brick2 Brick3: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick2 6. on mount point few files are missing [root@localhost rtest]# ls d25/filenew25 ls: d25/filenew25: No such file or directory [ this was present on mount point, check in step 3][root@rhsauto018 rpm]# ls -l /rhs/brick10/d25/filenew25 -rw-r--r-- 2 root root 0 May 29 15:16 /rhs/brick10/d25/filenew25 7. verify on all bricks, (including removed brick) in order to find this file present brick [root@rhsauto018 rpm]# ls -l /rhs/brick2/d25/filenew25 ls: cannot access /rhs/brick2/d25/filenew25: No such file or directory [root@rhsauto031 ~]# ls -l /rhs/brick2/d25/filenew25 ls: cannot access /rhs/brick2/d25/filenew25: No such file or directory [root@rhsauto038 rpm]# ls -l /rhs/brick2/d25/filenew25 ls: cannot access /rhs/brick2/d25/filenew25: No such file or directory removed - brick [root@rhsauto018 rpm]# ls -l /rhs/brick10/d25/filenew25 -rw-r--r-- 2 root root 0 May 29 15:16 /rhs/brick10/d25/filenew25 Actual results: files are missing Expected results: files should be present if remove-brick status says completed without any failures Additional info: more missing files :- [root@localhost rtest]# ls d19/filenew9 ls: d19/filenew9: No such file or directory [root@localhost rtest]# d24/filenew23 -bash: d24/filenew23: No such file or directory [root@localhost rtest]# d27/filenew34 -bash: d27/filenew34: No such file or directory [root@localhost rtest]# d27/filenew35 -bash: d27/filenew35: No such file or directory [root@localhost rtest]# d27/filenew36 -bash: d27/filenew36: No such file or directory
Along with comment #2, I have another question, does the single patch is the proposed fix or there would be more patches?
Lala, The fix currently will show a message upon remove-brick commit saying admin should check the decommissioned bricks for any files that might not have been migrated as part of rebalance. This patch is not a complete fix. Will open a new bug for the remove-brick status part. And this patch is applicable for 3.{4,5,6}.
Opened a new bug here https://bugzilla.redhat.com/show_bug.cgi?id=1136702.
The solution is to tell the admin to fix it while leaving the volume in a broken state? That's unacceptable. We need a process that leaves the files *in* the volume in the event the migration fails and *then* informs the admin of the failure so the admin can find the cause and fix it in order to try again.
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report. glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user