Description of problem: DHT - rebalance - 'gluster volume rebalance <volname> status' says 'not started' for all node but 'Rebalanced-files' and 'run time in secs' shows count and even log says its completed and it has actually migrated data. Version-Release number of selected component (if applicable): 3.4.0.7rhs-1.el6rhs.x86_64 How reproducible: Steps to Reproduce: [root@fred glusterd]# gluster volume status t1 Status of volume: t1 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick fan.lab.eng.blr.redhat.com:/rhs/brick1/t1 49155 Y 23093 Brick mia.lab.eng.blr.redhat.com:/rhs/brick1/t1 49155 Y 23630 Brick fred.lab.eng.blr.redhat.com:/rhs/brick1/t1 49153 Y 15263 NFS Server on localhost 2049 Y 15273 NFS Server on fdcb0533-eeb3-4054-8265-26558e92e65a 2049 Y 23104 NFS Server on d665808d-a42a-4eac-bf05-ca53c595486d 2049 Y 23641 There are no active volume tasks [root@fred ~]# gluster volume add-brick t1 fred.lab.eng.blr.redhat.com:/rhs/brick1/t2 volume add-brick: success [root@fred glusterd]# gluster volume rebalance t1 start force volume rebalance: t1: failed: Another transaction is in progress. Please try again after sometime. [root@fred glusterd]# gluster volume rebalance t1 start force volume rebalance: t1: success: Starting rebalance on volume t1 has been successful. ID: d87dac2b-5f8f-48db-ada4-c0346a86bc7c [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 0 not started 1.00 volume rebalance: t1: success: [root@fred glusterd]# gluster volume rebalance t1 status Node Rebalanced-files size scanned failures status run time in secs --------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 19 0Bytes 66 0 not started 1.00 fan.lab.eng.blr.redhat.com 26 0Bytes 121 0 not started 1.00 mia.lab.eng.blr.redhat.com 21 0Bytes 122 check the log <snip> [2013-05-14 14:38:51.091007] I [dht-rebalance.c:872:dht_migrate_file] 0-t1-dht: completed migration of /renamefile40 from subvolume t1- client-2 to t1-client-0 [2013-05-14 14:38:51.095263] I [dht-rebalance.c:663:dht_migrate_file] 0-t1-dht: /renamefile43: attempting to move from t1-client-2 to t 1-client-0 [2013-05-14 14:38:51.118271] I [dht-rebalance.c:872:dht_migrate_file] 0-t1-dht: completed migration of /renamefile43 from subvolume t1- client-2 to t1-client-0 [2013-05-14 14:38:51.122528] I [dht-rebalance.c:663:dht_migrate_file] 0-t1-dht: /renamefile45: attempting to move from t1-client-2 to t 1-client-1 [2013-05-14 14:38:51.155628] I [dht-rebalance.c:872:dht_migrate_file] 0-t1-dht: completed migration of /renamefile45 from subvolume t1- client-2 to t1-client-1 [2013-05-14 14:38:51.160429] I [dht-rebalance.c:663:dht_migrate_file] 0-t1-dht: /renamefile49: attempting to move from t1-client-2 to t 1-client-1 [2013-05-14 14:38:51.193842] I [dht-rebalance.c:872:dht_migrate_file] 0-t1-dht: completed migration of /renamefile49 from subvolume t1- client-2 to t1-client-1 [2013-05-14 14:38:51.207934] I [dht-rebalance.c:1311:gf_defrag_migrate_data] 0-t1-dht: Migration operation on dir / took 0.69 secs [2013-05-14 14:38:51.219623] I [dht-rebalance.c:1733:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs [2013-05-14 14:38:51.219655] I [dht-rebalance.c:1736:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 0, lookups: 66, failu res: 0 [2013-05-14 14:38:51.220247] W [glusterfsd.c:1011:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3296ce890d] (-->/lib64/libpthread.so.0() [0x3297407851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x40528d]))) 0-: received signum (15), shutting down Actual results: Expected results: Additional info: