Bug 1083115

Summary: DHT :- wrong 'missing' count in dht_layout_normalize anomalies log
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachana Patel <racpatel>
Component: distributeAssignee: Raghavendra G <rgowdapp>
Status: CLOSED CURRENTRELEASE QA Contact: RajeshReddy <rmekala>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.0CC: mzywusko, nbalacha, smohan
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: triaged
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-08 07:01:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rachana Patel 2014-04-01 13:52:17 UTC 
Description of problem:
mount log is full of log messages as below
[2014-03-24 11:53:24.115450] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir95. holes=2 overlaps=1 missing=1 down=0 misc=0

where holes, overlap and missing count is wrong.
(bug for wrong overlap and holes count exist)

Verified on back end and dir on all brick has hash layout, nothing is missing and all bricks were up

Version-Release number of selected component (if applicable):
3.4.0.59rhs-1.el6rhs.x86_64

How reproducible:
haven't tried

Steps to Reproduce:
Don't know exact steps to reproduce, but on lookup from mount point it was logged, all Directories were up and all brick had hash layout for that Directory.


[root@OVM4 ~]# gluster volume status down
Status of volume: down
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick 10.70.35.153:/rhs/brick1/n1			49152	Y	22114
Brick 10.70.35.153:/rhs/brick1/n2			49153	Y	27281
Brick 10.70.35.212:/rhs/brick1/n1			49153	Y	20940
Brick 10.70.35.212:/rhs/brick1/n2			49154	Y	20831
Brick 10.70.35.212:/rhs/brick1/n11			49155	Y	21810
Brick 10.70.35.212:/rhs/brick1/n12			49156	Y	22157
NFS Server on localhost					2049	Y	22177
NFS Server on 10.70.35.228				2049	Y	13939
NFS Server on 10.70.35.187				2049	Y	24144
NFS Server on 10.70.35.212				2049	Y	22169
 



Brick6: 10.70.35.212:/rhs/brick1/n12
[root@OVM3 ~]# getfattr -d -m . -e hex /rhs/brick1/n*/ndir90
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/n11/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x00000001000000002aaaaaaa55555553

# file: rhs/brick1/n12/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x0000000100000000555555547ffffffd

# file: rhs/brick1/n1/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x0000000100000000d5555552ffffffff

# file: rhs/brick1/n2/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x0000000100000000000000002aaaaaa9

[root@OVM4 ~]# getfattr -d -m . -e hex /rhs/brick1/n*/ndir90
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/n1/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x00000001000000007ffffffeaaaaaaa7

# file: rhs/brick1/n2/ndir90
trusted.gfid=0x061cfeba05154f6db6b7b0291ddc0f71
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8d5555551




Actual results:
on lookup, for few directories it shows wrong missing count

Expected results:
when hash range is complete and directory on all bricks has hash layout, missing count should be zero

Additional info:
log snippet

...
[2014-03-24 11:53:23.783680] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir51. holes=3 overlaps=0 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.800763] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir53. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.805614] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /n54. holes=2 overlaps=1 missing=
1 down=0 misc=0
[2014-03-24 11:53:23.817027] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir54. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.830966] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew56. holes=3 overlaps=0 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.867778] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew59. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.877362] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew60. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.886749] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew62. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.891455] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir62. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.897023] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew63. holes=3 overlaps=0 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.904631] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /ndir63. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.917952] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew65. holes=2 overlaps=1 missi
ng=1 down=0 misc=0
[2014-03-24 11:53:23.926919] I [dht-layout.c:646:dht_layout_normalize] 2-down-dht: found anomalies in /nnew66. holes=2 overlaps=1 missi
ng=1 down=0 misc=0