Bug 1537457 - DHT log messages: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
Summary: DHT log messages: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-0...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Nithya Balachandran
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1539516
TreeView+ depends on / blocked
 
Reported: 2018-01-23 09:16 UTC by Nithya Balachandran
Modified: 2019-02-12 03:25 UTC (History)
4 users (show)

Fixed In Version: glusterfs-v4.1.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1539516 (view as bug list)
Environment:
Last Closed: 2018-06-20 17:58:10 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Nithya Balachandran 2018-01-23 09:16:24 UTC
Description of problem:

Messages like the following are repeated multiple times in the client log files (fuse mount) for volumes with a single distribute subvolume (1xn volumes)

[2018-01-15 09:45:41.066914] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-gv0-dht:Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0




RCA:

If the dht subvol count is 1, dht_readdirp_cbk calls dht_populate_inode_for_dentry for each dir entry returned. This function tries to save the layout in inode. However, there is no layout on disk or gfid available for the '..' entry returned when performing an ls -l on the root of the subvol, causing the function to print this message.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Worker Ant 2018-01-23 09:37:33 UTC
REVIEW: https://review.gluster.org/19292 (cluster/dht: Skip '..' for the volume root dir) posted (#1) for review on master by N Balachandran

Comment 2 Nithya Balachandran 2018-01-23 09:38:43 UTC
Steps to Reproduce:
1. Create a volume with a single brick
2. Fuse mount the volume and perform an ls -l on the root of the volume
3. Check the mount logs

Comment 3 Worker Ant 2018-01-24 11:11:39 UTC
COMMIT: https://review.gluster.org/19292 committed in master by \"N Balachandran\" <nbalacha@redhat.com> with a commit message- cluster/dht: Skip '..' for the volume root dir

dht_populate_inode_for_dentry tries to update the layout
for the '..' entry when listing the root of the volume.
This entry does not correspond to an entry in the volume
and therefore does not have a gfid or a layout on disk,
causing layout processing to fail.

Change-Id: I2b7470e1c5e20d87b5545160697f24d041045140
BUG: 1537457
Signed-off-by: N Balachandran <nbalacha@redhat.com>

Comment 4 Worker Ant 2018-02-08 15:15:07 UTC
REVIEW: https://review.gluster.org/19529 (cluster/dht: Skip '..' for the volume root dir) posted (#1) for review on release-3.12 by N Balachandran

Comment 5 Worker Ant 2018-02-08 15:17:26 UTC
REVISION POSTED: https://review.gluster.org/19529 (cluster/dht: Skip '..' for the volume root dir) posted (#2) for review on release-3.12 by N Balachandran

Comment 6 nh2 2018-05-02 15:51:56 UTC
Should the fix for this that is in release-3.12 also have been applied to master?

I'm seeing this message in 4.0.

Comment 7 Nithya Balachandran 2018-05-03 09:10:27 UTC
(In reply to nh2 from comment #6)
> Should the fix for this that is in release-3.12 also have been applied to
> master?
> 
> I'm seeing this message in 4.0.

It looks like it has not made it into 4.0. 

I'm not sure if we will be having another 4.0.x release. It should, however, be there in 4.1.

Comment 8 nh2 2018-05-03 12:38:46 UTC
What I meant is: Is the commit in master? I couldn't find it.

Comment 9 Nithya Balachandran 2018-05-03 14:01:10 UTC
(In reply to nh2 from comment #8)
> What I meant is: Is the commit in master? I couldn't find it.

Yes, it is. See https://review.gluster.org/#/c/19292/

Comment 10 nh2 2018-05-03 14:02:29 UTC
Ah great, thanks.

Comment 11 Shyamsundar 2018-06-20 17:58:10 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.

glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 12 Artem Russakovskii 2019-02-12 00:52:49 UTC
Has this actually gotten fixed? I'm still seeing tons of these in 5.3.

[2019-02-12 00:50:22.373198] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0
[2019-02-12 00:50:22.373454] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0
[2019-02-12 00:50:22.373545] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0
[2019-02-12 00:50:22.373595] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0
[2019-02-12 00:50:22.373666] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0

Comment 13 Nithya Balachandran 2019-02-12 03:25:34 UTC
(In reply to Artem Russakovskii from comment #12)
> Has this actually gotten fixed? I'm still seeing tons of these in 5.3.
> 
> [2019-02-12 00:50:22.373198] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = ab1a38fe-33ca-4938-a9e8-48c3b26eec47). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373454] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = 03dd4702-39ad-476f-be8f-8301d5c07dc1). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373545] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = fef9cbce-fab5-405d-b330-ee7534cc3e61). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373595] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = ba968ad0-a179-4cd7-8269-5de001b43725). Holes=1 overlaps=0
> [2019-02-12 00:50:22.373666] I [MSGID: 109063]
> [dht-layout.c:659:dht_layout_normalize] 4-<SNIP>-dht: Found anomalies in
> (null) (gfid = 78733267-94a1-4493-b001-61419601d325). Holes=1 overlaps=0

The bug was for the null gfid being printed : (gfid = 00000000-0000-0000-0000-000000000000). A null gfid is an invalid gfid.

The messages you see are perfectly normal if there is a problem with the layout on those directories or if a brick is down.


Note You need to log in before you can comment on or make changes to this bug.