Bug 798658

Summary:	gluster volume rebalance force not moving the data
Product:	[Community] GlusterFS	Reporter:	M S Vishwanath Bhat <vbhat>
Component:	unclassified	Assignee:	shishir gowda <sgowda>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	pre-release	CC:	gluster-bugs, mzywusko, nsathyan, sgowda
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-03-08 11:43:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description M S Vishwanath Bhat 2012-02-29 14:22:43 UTC

Description of problem:
Created a single node distribute volume. Pushed some data to the mountpoint. Now added one more brick and ran rebalance. Files got hashed to other subvolume but data was still present in old one. Now even if I ran rebalance force, it's not actually moving the data to other subvolume. 

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa24

How reproducible:
1/1

Steps to Reproduce:
1. Create a dist volume with only one node. 
2. Create some data on mountpoint like untarring the linux kernel.
3. Add one more brick to volume.
4. Now run rebalance. Files got hashed to other subvolume, but data present in first subvolume. 
5. Now ran rebalance force.

  
Actual results:
Data still present in first subvolume, even though both bricks had around 40GB free available space.

Expected results:
Some of the data moved to other subvolume also. 

Additional info:

Entries from rebalance logs. 


[2012-02-29 05:38:39.484023] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/usr/gen_init_cpio.c
[2012-02-29 05:38:39.485082] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485121] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485149] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486924] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486962] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486989] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.488154] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.489244] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.492841] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.494008] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.497111] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/Kconfig: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.498728] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/Kconfig)
[2012-02-29 05:38:39.498768] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/Kconfig
[2012-02-29 05:38:39.501406] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/async_pf.h: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.503093] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/async_pf.h)
[2012-02-29 05:38:39.503133] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/async_pf.h
[2012-02-29 05:38:39.504910] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.506590] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c)
[2012-02-29 05:38:39.506630] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c
[2012-02-29 05:38:39.510739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510780] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510808] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513080] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513118] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513154] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515672] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515711] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515770] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515806] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515833] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.517560] I [dht-rebalance.c:1516:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 36783
[2012-02-29 05:38:39.566693] W [glusterfsd.c:783:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3613ee577d] (-->/lib64/libpthread.so.0() [0x36146077e1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x40747d]))) 0-: received signum (15), shutting down



I have attached rebalance log from one machine. Have archived other logs.

Comment 1 M S Vishwanath Bhat 2012-02-29 14:24:48 UTC

It seems log file couldn't be attached because it was too big (27MB). Have archived it.

Comment 2 shishir gowda 2012-03-01 04:26:41 UTC

On the latest githead, I am not able to reproduce this issue.
Data migration happens successfully when force option is given.
Please provide more details, or try to reproduce the bug.

Comment 3 M S Vishwanath Bhat 2012-03-02 14:29:21 UTC

I was able to reproduce this once more in my set-up. Saving all the logs.

Comment 4 shishir gowda 2012-03-08 03:55:00 UTC

This seems to be a duplicate of bug 799887.

Can you please confirm if the bug is now fixed?

Comment 5 M S Vishwanath Bhat 2012-03-08 09:49:39 UTC

(In reply to comment #4)
> This seems to be a duplicate of bug 799887.
> 
> Can you please confirm if the bug is now fixed?

I just tried with the latest git pull. But the issue isn't fixed.

Comment 6 M S Vishwanath Bhat 2012-03-08 11:43:26 UTC

With parallel rebalance fixes in latest git pull, rebalance force is moving the data properly. And after rebalance is completed, I don't see any files with sticky bit set, in the back-end export directory.