Description of problem: mount log is full of log messages as below [2014-03-24 11:53:24.115450] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir95. holes=2 overlaps=1 missing=1 down=0 misc=0 where holes, overlap and missing count is wrong. (bug for wrong overlap and holes count exist) Verified on back end and dir on all brick has hash layout, nothing is missing and all bricks were up Version-Release number of selected component (if applicable): 3.4.0.59rhs-1.el6rhs.x86_64 How reproducible: haven't tried Steps to Reproduce: Don't know exact steps to reproduce, but on lookup from mount point it was logged, all Directories were up and all brick had hash layout for that Directory. [root@OVM4 ~]# gluster volume status down Status of volume: down Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.153:/rhs/brick1/n1 49152 Y 22114 Brick 10.70.35.153:/rhs/brick1/n2 49153 Y 27281 Brick 10.70.35.212:/rhs/brick1/n1 49153 Y 20940 Brick 10.70.35.212:/rhs/brick1/n2 49154 Y 20831 Brick 10.70.35.212:/rhs/brick1/n11 49155 Y 21810 Brick 10.70.35.212:/rhs/brick1/n12 49156 Y 22157 NFS Server on localhost 2049 Y 22177 NFS Server on 10.70.35.228 2049 Y 13939 NFS Server on 10.70.35.187 2049 Y 24144 NFS Server on 10.70.35.212 2049 Y 22169 Brick6: 10.70.35.212:/rhs/brick1/n12 [root@OVM3 ~]# getfattr -d -m . -e hex /rhs/brick1/n*/ndir90 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/n11/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553 # file: rhs/brick1/n12/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x0000000100000000555555547ffffffd # file: rhs/brick1/n1/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff # file: rhs/brick1/n2/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9 [root@OVM4 ~]# getfattr -d -m . -e hex /rhs/brick1/n*/ndir90 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/n1/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7 # file: rhs/brick1/n2/ndir90 trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71 trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551 Actual results: on lookup, for few directories it shows wrong missing count Expected results: when hash range is complete and directory on all bricks has hash layout, missing count should be zero Additional info: log snippet ... [2014-03-24 11:53:23.783680] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir51. holes=3 overlaps=0 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.800763] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir53. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.805614] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /n54. holes=2 overlaps=1 missing= 1 down=0 misc=0 [2014-03-24 11:53:23.817027] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir54. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.830966] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew56. holes=3 overlaps=0 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.867778] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew59. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.877362] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew60. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.886749] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew62. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.891455] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir62. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.897023] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew63. holes=3 overlaps=0 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.904631] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir63. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.917952] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew65. holes=2 overlaps=1 missi ng=1 down=0 misc=0 [2014-03-24 11:53:23.926919] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew66. holes=2 overlaps=1 missi ng=1 down=0 misc=0