Description of problem: Created a single node distribute volume. Pushed some data to the mountpoint. Now added one more brick and ran rebalance. Files got hashed to other subvolume but data was still present in old one. Now even if I ran rebalance force, it's not actually moving the data to other subvolume. Version-Release number of selected component (if applicable): glusterfs-3.3.0qa24 How reproducible: 1/1 Steps to Reproduce: 1. Create a dist volume with only one node. 2. Create some data on mountpoint like untarring the linux kernel. 3. Add one more brick to volume. 4. Now run rebalance. Files got hashed to other subvolume, but data present in first subvolume. 5. Now ran rebalance force. Actual results: Data still present in first subvolume, even though both bricks had around 40GB free available space. Expected results: Some of the data moved to other subvolume also. Additional info: Entries from rebalance logs. [2012-02-29 05:38:39.484023] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/usr/gen_init_cpio.c [2012-02-29 05:38:39.485082] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.485121] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.485149] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.486924] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.486962] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.486989] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.488154] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt [2012-02-29 05:38:39.489244] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt [2012-02-29 05:38:39.492841] I [dht-common.c:2297:dht_setxattr] 0-hosdu-dht: fixing the layout of /kernel-source/linux-3.0.1/virt/kvm [2012-02-29 05:38:39.494008] I [dht-rebalance.c:1045:gf_defrag_migrate_data] 0-hosdu-dht: migate data called on /kernel-source/linux-3.0.1/virt/kvm [2012-02-29 05:38:39.497111] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/Kconfig: attempting to move from hosdu-client-0 to hosdu-client-1 [2012-02-29 05:38:39.498728] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/Kconfig) [2012-02-29 05:38:39.498768] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/Kconfig [2012-02-29 05:38:39.501406] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/async_pf.h: attempting to move from hosdu-client-0 to hosdu-client-1 [2012-02-29 05:38:39.503093] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/async_pf.h) [2012-02-29 05:38:39.503133] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/async_pf.h [2012-02-29 05:38:39.504910] I [dht-rebalance.c:630:dht_migrate_file] 0-hosdu-dht: /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c: attempting to move from hosdu-client-0 to hosdu-client-1 [2012-02-29 05:38:39.506590] W [dht-rebalance.c:355:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-client-0) with higher disk space to a node (hosdu-client-1) with lesser disk space (/kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c) [2012-02-29 05:38:39.506630] E [dht-rebalance.c:1141:gf_defrag_migrate_data] 0-hosdu-dht: setxattr failed for /kernel-source/linux-3.0.1/virt/kvm/coalesced_mmio.c [2012-02-29 05:38:39.510739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.510780] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.510808] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.513080] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.513118] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.513154] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515672] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515711] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515739] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515770] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515806] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.515833] I [dict.c:339:dict_get] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x211) [0x7f5828112919] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/client.so(client3_1_readdirp_cbk+0x345) [0x7f581fdddee7] (-->/usr/local/lib/glusterfs/3git/xlator/cluster/distribute.so(dht_readdirp_cbk+0x1dd) [0x7f581fb9526c]))) 0-dict: !this || key=trusted.glusterfs.dht.linkto [2012-02-29 05:38:39.517560] I [dht-rebalance.c:1516:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 36783 [2012-02-29 05:38:39.566693] W [glusterfsd.c:783:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3613ee577d] (-->/lib64/libpthread.so.0() [0x36146077e1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x40747d]))) 0-: received signum (15), shutting down I have attached rebalance log from one machine. Have archived other logs.
It seems log file couldn't be attached because it was too big (27MB). Have archived it.
On the latest githead, I am not able to reproduce this issue. Data migration happens successfully when force option is given. Please provide more details, or try to reproduce the bug.
I was able to reproduce this once more in my set-up. Saving all the logs.
This seems to be a duplicate of bug 799887. Can you please confirm if the bug is now fixed?
(In reply to comment #4) > This seems to be a duplicate of bug 799887. > > Can you please confirm if the bug is now fixed? I just tried with the latest git pull. But the issue isn't fixed.
With parallel rebalance fixes in latest git pull, rebalance force is moving the data properly. And after rebalance is completed, I don't see any files with sticky bit set, in the back-end export directory.