Bug 975364 - DHT : remove-brick : - remove-brick migrates file from brick/sub-volume which is not removed (without running rebalance)
DHT : remove-brick : - remove-brick migrates file from brick/sub-volume which...
Status: CLOSED NOTABUG
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: shishir gowda
amainkar
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-18 05:08 EDT by Rachana Patel
Modified: 2015-04-20 07:56 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-05 02:01:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
server log (6.91 MB, application/x-xz)
2013-06-18 05:08 EDT, Rachana Patel
no flags Details

  None (edit)
Description Rachana Patel 2013-06-18 05:08:37 EDT
Created attachment 762420 [details]
server log

Description of problem:
DHT : remove-brick : - remove-brick migrates file from brick/sub-volume which is not removed (without running rebalance) 

Version-Release number of selected component (if applicable):
3.4.0.9rhs-1.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. created a volume having 4 sub-vol on 2 RHS node
[root@cutlass ~]#gluster volume create onqa cutlass.lab.eng.blr.redhat.com:/rhs/brick1/o1 mia.lab.eng.blr.redhat.com:/rhs/brick1/o2  cutlass.lab.eng.blr.redhat.com:/rhs/brick1/o2 mia.lab.eng.blr.redhat.com:/rhs/brick1/o1
volume create: onqa: success: please start the volume to access data
[root@cutlass ~]# gluster volume start onqa
volume start: onqa: success

2. fuse mount that volume and create few files and dir on it
[root@rhsauto037 onqa]# mount -t glusterfs cutlass.lab.eng.blr.redhat.com:/onqa /mnt/onqa
[root@rhsauto037 onqa]# cd /mnt/onqa
[root@rhsauto037 onqa]# for i in {1..20}; do mkdir d$i;  touch f"$i" ; touch d1/f"$i"; done
[root@rhsauto037 onqa]# ls
d1   d11  d13  d15  d17  d19  d20  d4  d6  d8  f1   f11  f13  f15  f17  f19  f20  f4  f6  f8
d10  d12  d14  d16  d18  d2   d3   d5  d7  d9  f10  f12  f14  f16  f18  f2   f3   f5  f7  f9

3.verify hash-layout for that file and list files.
server1:-

[root@cutlass ~]# getfattr -d -m . -e hex /rhs/brick1/o*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/o1
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

# file: rhs/brick1/o2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

[root@cutlass ~]# ls -l /rhs/brick1/o*/f*
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f12
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f17
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f4
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f8
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f11
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f14
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f15
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f3
-rw-r--r-- 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f5


server-2:-
[root@mia ~]# getfattr -d -m . -e hex /rhs/brick1/o*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/o1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

# file: rhs/brick1/o2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

[root@mia ~]# ls -l /rhs/brick1/o*/f*
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f1
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f10
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f18
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f19
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f2
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o1/f20
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f13  <-----------------
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f16
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f6
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f7
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f9

4. remove-brick with start option
[root@cutlass ~]# gluster volume remove-brick onqa mia.lab.eng.blr.redhat.com:/rhs/brick1/o1 status                                    Node Rebalanced-files          size       scanned      failures         status run-time in secs
                               ---------      -----------   -----------   -----------   -----------   ------------   --------------
                               localhost                0        0Bytes             0             0    not started             0.00
              fan.lab.eng.blr.redhat.com                0        0Bytes             0             0    not started             0.00
             fred.lab.eng.blr.redhat.com                0        0Bytes             0             0    not started             0.00
              mia.lab.eng.blr.redhat.com                0        0Bytes             0             0    not started             0.00
[root@cutlass ~]# gluster volume remove-brick onqa mia.lab.eng.blr.redhat.com:/rhs/brick1/o1 start
volume remove-brick start: success
ID: 3de17309-9f00-422a-80a9-1c0ef6dd4aa2
[root@cutlass ~]# gluster volume remove-brick onqa mia.lab.eng.blr.redhat.com:/rhs/brick1/o1 status
                                    Node Rebalanced-files          size       scanned      failures         status run-time in secs
                               ---------      -----------   -----------   -----------   -----------   ------------   --------------
                               localhost                0        0Bytes             0             0    not started             0.00
              fan.lab.eng.blr.redhat.com                0        0Bytes             0             0    not started             0.00
             fred.lab.eng.blr.redhat.com                0        0Bytes             0             0    not started             0.00
              mia.lab.eng.blr.redhat.com               12        0Bytes            40             0      completed             0.00


5. when remove-brick status is completed verify hash layout of bricks and also check that files should be migrated from removed-brick only

server1:-
[root@cutlass ~]# getfattr -d -m . -e hex /rhs/brick1/o*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/o1
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000000000000055555554
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

# file: rhs/brick1/o2
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

[root@cutlass ~]# ls -l /rhs/brick1/o*/f*
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f12
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f13  <----------------
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f17
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f4
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o1/f8
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f1
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f10
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f11
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f14
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f15
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f18
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f19
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f2
-rw-r--r--. 2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f20
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f3
-rw-r--r--  2 root root 0 Jun 17 20:14 /rhs/brick1/o2/f5

server 2 :-
[root@mia ~]# getfattr -d -m . -e hex /rhs/brick1/o*
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/o1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000000000000000000000
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

# file: rhs/brick1/o2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9
trusted.glusterfs.volume-id=0x9e1be9e55a5a41cab46dd520cf09372e

[root@mia ~]# ls -l /rhs/brick1/o*/f*
---------T. 2 root root 0 Jun 18 05:23 /rhs/brick1/o2/f11
---------T. 2 root root 0 Jun 18 05:23 /rhs/brick1/o2/f14
---------T. 2 root root 0 Jun 18 05:23 /rhs/brick1/o2/f15
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f16
---------T. 2 root root 0 Jun 18 05:23 /rhs/brick1/o2/f3
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f6
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f7
-rw-r--r--. 2 root root 0 Jun 18  2013 /rhs/brick1/o2/f9


Actual results:
file f13 got migrated from server mia(/rhs/brick1/o2) to cutlass(/rhs/brick1/o1) without running rebalance

Expected results:
In case of remove-brick 'start', It should migrate files from the brick which is being removed.

Additional info:
Comment 4 Amar Tumballi 2013-06-19 23:09:22 EDT
Shishir, is it the same issue you fixed with layout?
Comment 5 shishir gowda 2013-07-05 02:01:18 EDT
When remove-brick operation is started, the layouts of the bricks get re-distributed. The bricks which have been decommissioned(being removed) get a layout of "0x00000".

The decision of whether to migrate a file is based on the hash value of the file, and hash range of the brick it resides on. For a decommissioned brick, since the layout is zeroed out, the migration is guaranteed.

But for other files, migration is not guaranteed, but might migrate as the layouts are now re-written.

In effect, files have to be migrated away from the bricks that are being decommissioned, and due to layout re-writes other files from other bricks might also end up being migrated.

Closing the bug, as it works as intended.

Please re-open the bug if my understanding of the bug is incorrect.

Note You need to log in before you can comment on or make changes to this bug.