Description of problem: rebalance log has errors '[2013-11-23 03:17:58.070608] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/f806 gfid not present' [2013-11-23 03:17:58.079225] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc1 gfid not present [2013-11-23 03:17:58.079266] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc2 gfid not present [2013-11-23 03:17:58.079279] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc3 gfid not present [2013-11-23 03:17:58.079290] E [dht-rebalance.c:1191:gf_defrag_migrate_data] 0-flat-dht: /mvs1/mvetc4 gfid not present for different Directories and files around 980 errors. [root@7-VM1 core]# grep 'gfid not present' /var/log/glusterfs/flat-rebalance.log-20131124 | wc -l 980 In case of Directories, on all sub-volumes, gfid is set and same In case of file, gfid is present on one sub-volume. Then why so many errors? e.g. file:- [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick2/f/mvs1/f806 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/f/mvs1/f806 trusted.gfid=0x2a74c64b31b149eca397466b783898b8 Directory:- [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc4 trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick2/f/mvs1/mvetc4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/f/mvs1/mvetc4 trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e trusted.glusterfs.dht=0x0000000100000000000000003ffffffe [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick4/f/mvs1/mvetc4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick4/f/mvs1/mvetc4 trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e trusted.glusterfs.dht=0x0000000100000000ccccccccffffffff [root@7-VM1 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc4 trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd [root@7-VM3 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc4 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc4 trusted.gfid=0x200ecbd616694d8180985c3f68d3d86e trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc another dir:- [root@7-VM1 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc3 trusted.gfid=0xc0386717b0a64231aa83d260db2670cd trusted.glusterfs.dht=0x00000001000000003fffffff7ffffffd [root@7-VM3 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc3 trusted.gfid=0xc0386717b0a64231aa83d260db2670cd trusted.glusterfs.dht=0x00000001000000007ffffffebffffffc [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick4/f/mvs1/mvetc3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick4/f/mvs1/mvetc3 trusted.gfid=0xc0386717b0a64231aa83d260db2670cd trusted.glusterfs.dht=0x00000001000000003333333366666665 [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick2/f/mvs1/mvetc3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick2/f/mvs1/mvetc3 trusted.gfid=0xc0386717b0a64231aa83d260db2670cd trusted.glusterfs.dht=0x0000000100000000000000003ffffffe [root@7-VM4 ~]# getfattr -d -m . -e hex /rhs/brick1/f/mvs1/mvetc3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/f/mvs1/mvetc3 trusted.gfid=0xc0386717b0a64231aa83d260db2670cd trusted.glusterfs.dht=0x0000000100000000bffffffdffffffff Version-Release number of selected component (if applicable): ============================================ 3.4.0.44rhs-1.el6rhs.x86_64 How reproducible: ================== haven't tried Steps to Reproduce: ==================== 1. create and mount DHT volume. Create Data from mount point(Directory depth was 10) 2.add brick to volume and start rebalance. 3. while rebalance is in progress, perform rename operation for directories and files 3. after 44+ hours rebalance process was crashed on all node and rebalance status was 'failed' 4. checked rebalance log and found this errors. Actual results: showing errors in log related to gfid Expected results: if gfid is present then it should not show errors in log Additional info:
Cloning this to 3.1. to be fixed in future release.