Description of problem:
Created a single node distribute volume. Pushed some data to the mountpoint. Now added one more brick and ran rebalance. Files got hashed to other subvolume but data was still present in old one. Now even if I ran rebalance force, it's not actually moving the data to other subvolume. 

Version-Release number of selected component (if applicable):
glusterfs-3.3.0qa24

How reproducible:
1/1

Steps to Reproduce:
1. Create a dist volume with only one node. 
2. Create some data on mountpoint like untarring the linux kernel.
3. Add one more brick to volume.
4. Now run rebalance. Files got hashed to other subvolume, but data present in first subvolume. 
5. Now ran rebalance force.

  
Actual results:
Data still present in first subvolume, even though both bricks had around 40GB free available space.

Expected results:
Some of the data moved to other subvolume also. 

Additional info:

Entries from rebalance logs. 


[2012-02-29 05:38:39.484023] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/usr/gen_init_cpio.c
[2012-02-29 05:38:39.485082] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485121] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.485149] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486924] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486962] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.486989] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.488154] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.489244] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt
[2012-02-29 05:38:39.492841] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.494008] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt/kvm
[2012-02-29 05:38:39.497111] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/Kconfig: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.498728] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/Kconfig)
[2012-02-29 05:38:39.498768] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/Kconfig
[2012-02-29 05:38:39.501406] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/async_pf.h: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.503093] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/async_pf.h)
[2012-02-29 05:38:39.503133] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/async_pf.h
[2012-02-29 05:38:39.504910] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c: attempting to move from hosdu-client-0 to hosdu-client-1
[2012-02-29 05:38:39.506590] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c)
[2012-02-29 05:38:39.506630] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c
[2012-02-29 05:38:39.510739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510780] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.510808] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513080] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513118] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.513154] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515672] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515711] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515770] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515806] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.515833] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto
[2012-02-29 05:38:39.517560] I [dht-rebalance.c:1516:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 36783
[2012-02-29 05:38:39.566693] W [glusterfsd.c:783:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3613ee577d] (-->/lib64/libpthread.so.0() [0x36146077e1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x40747d]))) 0-: received signum (15), shutting down



I have attached rebalance log from one machine. Have archived other logs.