Description of problem: ======================= rebalance start force failed to migrate file with error ' gfid not present' but status shows 0 failure [2014-08-06 07:55:27.115723] E [MSGID: 109010] [dht-rebalance.c:1434:gf_defrag_migrate_data] 0-brick1-dht: //new gfid not present [root@OVM3 brick1]# gluster volume rebalance brick1 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 10.70.35.240 0 0Bytes 1 0 0 completed 0.00 10.70.35.172 0 0Bytes 1 0 0 completed 0.00 volume rebalance: brick1: success: Version-Release number of selected component (if applicable): ============================================================= 3.6.0.27-1.el6rhs.x86_64 How reproducible: ================ got twice work around:- kill all brick process and glusterd and start again. After that it migrated file Steps to Reproduce: ================== 1. had distributed volume. removed all data from mount point. Stop volume and kill all glusterd/glusterfsd processes. Upgrade gluster rpms 2. start gluster processes. Start volume and mount it again. 3. create one file and rename that file so hashed and cached sub-volumes are different. 4. start rebalance process and check data migration. rebalance status says no failures but log has migration error and file was not migrated [root@OVM3 brick1]# gluster volume rebalance brick1 status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 10.70.35.240 0 0Bytes 1 0 0 completed 0.00 10.70.35.172 0 0Bytes 1 0 0 completed 0.00 volume rebalance: brick1: success: brick:- [root@OVM4 brick1]# ls -l total 0 ---------T 2 root root 0 Aug 6 13:21 new [root@OVM4 brick1]# ls -l total 0 ---------T 2 root root 0 Aug 6 13:21 new Actual results: =============== - failure count is wrong Expected results: ================= - In case of failure status should show correct count Additional info: =============== log snippet:- [2014-08-06 07:55:27.112548] I [dht-rebalance.c:1356:gf_defrag_migrate_data] 0-brick1-dht: migrate data called on / [2014-08-06 07:55:27.115723] E [MSGID: 109010] [dht-rebalance.c:1434:gf_defrag_migrate_data] 0-brick1-dht: //new gfid not present [2014-08-06 07:55:27.116171] I [dht-rebalance.c:1596:gf_defrag_migrate_data] 0-brick1-dht: Migration operation on dir / took 0.00 secs [2014-08-06 07:55:27.119414] I [MSGID: 109028] [dht-rebalance.c:2058:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs [2014-08-06 07:55:27.119444] I [MSGID: 109028] [dht-rebalance.c:2062:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 1, failures: 0, skipped: 0