Bug 1537457

Summary:	DHT log messages: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
Product:	[Community] GlusterFS	Reporter:	Nithya Balachandran <nbalacha>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	archon810, bugs, nbalacha, nh2-redhatbugzilla
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-v4.1.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1539516 (view as bug list)		Environment:
Last Closed:	2018-06-20 17:58:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1539516

Description Nithya Balachandran 2018-01-23 09:16:24 UTC

Description of problem:

Messages like the following are repeated multiple times in the client log files (fuse mount) for volumes with a single distribute subvolume (1xn volumes)

[2018-01-15 09:45:41.066914] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-gv0-dht:Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0




RCA:

If the dht subvol count is 1, dht_readdirp_cbk calls dht_populate_inode_for_dentry for each dir entry returned. This function tries to save the layout in inode. However, there is no layout on disk or gfid available for the '..' entry returned when performing an ls -l on the root of the subvol, causing the function to print this message.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Worker Ant 2018-01-23 09:37:33 UTC

REVIEW: https://review.gluster.org/19292 (cluster/dht: Skip '..' for the volume root dir) posted (#1) for review on master by N Balachandran

Comment 2 Nithya Balachandran 2018-01-23 09:38:43 UTC

Steps to Reproduce:
1. Create a volume with a single brick
2. Fuse mount the volume and perform an ls -l on the root of the volume
3. Check the mount logs

Comment 3 Worker Ant 2018-01-24 11:11:39 UTC

COMMIT: https://review.gluster.org/19292 committed in master by \"N Balachandran\" <nbalacha> with a commit message- cluster/dht: Skip '..' for the volume root dir

dht_populate_inode_for_dentry tries to update the layout
for the '..' entry when listing the root of the volume.
This entry does not correspond to an entry in the volume
and therefore does not have a gfid or a layout on disk,
causing layout processing to fail.

Change-Id: I2b7470e1c5e20d87b5545160697f24d041045140
BUG: 1537457
Signed-off-by: N Balachandran <nbalacha>

Comment 4 Worker Ant 2018-02-08 15:15:07 UTC

REVIEW: https://review.gluster.org/19529 (cluster/dht: Skip '..' for the volume root dir) posted (#1) for review on release-3.12 by N Balachandran

Comment 5 Worker Ant 2018-02-08 15:17:26 UTC

REVISION POSTED: https://review.gluster.org/19529 (cluster/dht: Skip '..' for the volume root dir) posted (#2) for review on release-3.12 by N Balachandran

Comment 6 nh2 2018-05-02 15:51:56 UTC

Should the fix for this that is in release-3.12 also have been applied to master?

I'm seeing this message in 4.0.

Comment 7 Nithya Balachandran 2018-05-03 09:10:27 UTC

(In reply to nh2 from comment #6)
> Should the fix for this that is in release-3.12 also have been applied to
> master?
> 
> I'm seeing this message in 4.0.

It looks like it has not made it into 4.0. 

I'm not sure if we will be having another 4.0.x release. It should, however, be there in 4.1.

Comment 8 nh2 2018-05-03 12:38:46 UTC

What I meant is: Is the commit in master? I couldn't find it.

Comment 9 Nithya Balachandran 2018-05-03 14:01:10 UTC

(In reply to nh2 from comment #8)
> What I meant is: Is the commit in master? I couldn't find it.

Yes, it is. See https://review.gluster.org/#/c/19292/

Comment 10 nh2 2018-05-03 14:02:29 UTC

Ah great, thanks.

Comment 11 Shyamsundar 2018-06-20 17:58:10 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.

glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 12 Artem Russakovskii 2019-02-12 00:52:49 UTC

Has this actually gotten fixed? I'm still seeing tons of these in 5.3.

[2019-02-12 00:50:22.373198] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0
[2019-02-12 00:50:22.373454] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0
[2019-02-12 00:50:22.373545] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0
[2019-02-12 00:50:22.373595] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0
[2019-02-12 00:50:22.373666] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0

Comment 13 Nithya Balachandran 2019-02-12 03:25:34 UTC

(In reply to Artem Russakovskii from comment #12)
> Has this actually gotten fixed? I'm still seeing tons of these in 5.3.
> 
> [2019-02-12 00:50:22.373198] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373454] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373545] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373595] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373666] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0

The bug was for the null gfid being printed : (gfid = 00000000-0000-0000-0000-000000000000). A null gfid is an invalid gfid.

The messages you see are perfectly normal if there is a problem with the layout on those directories or if a brick is down.