Bug 798658 - gluster volume rebalance force not moving the data
gluster volume rebalance force not moving the data
Status: CLOSED UPSTREAM
Product: GlusterFS
Classification: Community
Component: unclassified (Show other bugs)
pre-release
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: shishir gowda
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-02-29 09:22 EST by M S Vishwanath Bhat
Modified: 2016-05-31 21:55 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-08 06:43:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description M S Vishwanath Bhat 2012-02-29 09:22:43 EST
Description of problem:
Created a single node distribute volume. Pushed some data to the mountpoint. Now added one more brick and ran rebalance. Files got hashed to other subvolume but data was still present in old one. Now even if I ran rebalance force, it's not actually moving the data to other subvolume. 

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa24

How reproducible:
1/1

Steps to Reproduce:
1. Create a dist volume with only one node. 
2. Create some data on mountpoint like untarring the linux kernel.
3. Add one more brick to volume.
4. Now run rebalance. Files got hashed to other subvolume, but data present in first subvolume. 
5. Now ran rebalance force.

  
Actual results:
Data still present in first subvolume, even though both bricks had around 40GB free available space.

Expected results:
Some of the data moved to other subvolume also. 

Additional info:

Entries from rebalance logs. 


[2012-02-29 05:38:39.484023] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/usr/gen_init_cpio.c
[2012-02-29 05:38:39.485082] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485121] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485149] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486924] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486962] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486989] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.488154] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.489244] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.492841] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.494008] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.497111] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/Kconfig: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.498728] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/Kconfig)
[2012-02-29 05:38:39.498768] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/Kconfig
[2012-02-29 05:38:39.501406] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/async_pf.h: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.503093] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/async_pf.h)
[2012-02-29 05:38:39.503133] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/async_pf.h
[2012-02-29 05:38:39.504910] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.506590] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c)
[2012-02-29 05:38:39.506630] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c
[2012-02-29 05:38:39.510739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510780] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510808] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513080] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513118] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513154] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515672] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515711] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515770] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515806] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515833] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.517560] I [dht-rebalance.c:1516:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 36783
[2012-02-29 05:38:39.566693] W [glusterfsd.c:783:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3613ee577d] (-->/lib64/libpthread.so.0() [0x36146077e1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x40747d]))) 0-: received signum (15), shutting down



I have attached rebalance log from one machine. Have archived other logs.
Comment 1 M S Vishwanath Bhat 2012-02-29 09:24:48 EST
It seems log file couldn't be attached because it was too big (27MB). Have archived it.
Comment 2 shishir gowda 2012-02-29 23:26:41 EST
On the latest githead, I am not able to reproduce this issue.
Data migration happens successfully when force option is given.
Please provide more details, or try to reproduce the bug.
Comment 3 M S Vishwanath Bhat 2012-03-02 09:29:21 EST
I was able to reproduce this once more in my set-up. Saving all the logs.
Comment 4 shishir gowda 2012-03-07 22:55:00 EST
This seems to be a duplicate of bug 799887.

Can you please confirm if the bug is now fixed?
Comment 5 M S Vishwanath Bhat 2012-03-08 04:49:39 EST
(In reply to comment #4)
> This seems to be a duplicate of bug 799887.
> 
> Can you please confirm if the bug is now fixed?

I just tried with the latest git pull. But the issue isn't fixed.
Comment 6 M S Vishwanath Bhat 2012-03-08 06:43:26 EST
With parallel rebalance fixes in latest git pull, rebalance force is moving the data properly. And after rebalance is completed, I don't see any files with sticky bit set, in the back-end export directory.

Note You need to log in before you can comment on or make changes to this bug.