Bug 1034699

Description of problem:
rebalance log has errors
'[2013-11-23 03:17:58.070608] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/f806 gfid not present'
[2013-11-23 03:17:58.079225] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc1 gfid not present
[2013-11-23 03:17:58.079266] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc2 gfid not present
[2013-11-23 03:17:58.079279] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc3 gfid not present
[2013-11-23 03:17:58.079290] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc4 gfid not present


for different Directories and files around 980 errors.

[root@7-VM1 core]# grep 'gfid not present' /var/log/glusterfs/flat-rebalance.log-20131124  | wc -l
980

In case of Directories, on all sub-volumes, gfid is set and same
In case of file, gfid is present on one sub-volume.

Then why so many errors?

e.g.

file:-
[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick2/f/mvs1/f806
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/f/mvs1/f806
trusted.gfid=0x2a74c64b31b149eca397466b783898b8

Directory:-

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc4
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc4
trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick2/f/mvs1/mvetc4
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/f/mvs1/mvetc4
trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick4/f/mvs1/mvetc4
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick4/f/mvs1/mvetc4
trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e
trusted.glusterfs.dht=0x0000000100000000ccccccccffffffff

[root@7-VM1 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc4
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc4
trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd

[root@7-VM3 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc4
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc4
trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc


another dir:-
[root@7-VM1 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc3
trusted.gfid=0xc0386717b0a64231aa83d260db2670cd
trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd
[root@7-VM3 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc3
trusted.gfid=0xc0386717b0a64231aa83d260db2670cd
trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick4/f/mvs1/mvetc3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick4/f/mvs1/mvetc3
trusted.gfid=0xc0386717b0a64231aa83d260db2670cd
trusted.glusterfs.dht=0x00000001000000003333333366666665

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick2/f/mvs1/mvetc3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/f/mvs1/mvetc3
trusted.gfid=0xc0386717b0a64231aa83d260db2670cd
trusted.glusterfs.dht=0x0000000100000000000000003ffffffe

[root@7-VM4 ~]# getfattr -d -m . -e hex   /rhs/brick1/f/mvs1/mvetc3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/f/mvs1/mvetc3
trusted.gfid=0xc0386717b0a64231aa83d260db2670cd
trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff



Version-Release number of selected component (if applicable):
============================================
3.4.0.44rhs-1.el6rhs.x86_64

How reproducible:
==================
haven't tried


Steps to Reproduce:
====================
1. create and mount DHT volume. Create Data from mount point(Directory depth was 10)
2.add brick to volume and start rebalance.
3. while rebalance is in progress, perform rename operation for directories and files 
3. after 44+ hours rebalance process was crashed on all node and rebalance status was 'failed'
4. checked rebalance log and found this errors.

Actual results:
showing errors in log related to gfid

Expected results:
if gfid is present then it should not show errors in log

Additional info: