Bug 1554509

(In reply to Shyamsundar from comment #0)
> Description of problem:
> 
> Upgraded a 3.12 3 servers + 1 client setup to a 4.0.0 version and noticed
> the following errors in the client mount logs,
> 
> NOTE: not sure if a fresh 4.0.0 setup also gives the same errors, but the
> upgrade case did.
> 
> 
> Post upgrade, client mount is seeing this error
> I [MSGID: 109005] [dht-selfheal.c:2328:dht_selfheal_directory] 0-patchy-dht:
> Directory selfheal failed: Unable to form layout for directory /
> 
> and, this message
> I [MSGID: 109063] [dht-layout.c:693:dht_layout_normalize] 0-patchy-dht:
> Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> Holes=1 overlaps=0

This is a false positive of an error. These errors are normally seen when a lookup races with mkdir and finds a directory with layout _yet_ to be set. IOW, its a transient condition which gets resolved once either mkdir or lookup completes.

> 
> Inspection of brick root on the servers, shows a proper layout that is fully
> formed,
> # getfattr -d  -e hex -m . /d/brick1     
> getfattr: Removing leading '/' from absolute path names
> # file: d/brick1
> security.
> selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733
> 000
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.patchy-client-1=0x000000000000000000000000
> trusted.afr.patchy-client-2=0x000000000000000000000000
> trusted.gfid=0x00000000000000000000000000000001
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
> trusted.glusterfs.volume-id=0xb43003c35dde4cf0915ac7d9ac94764e
> 
> Version-Release number of selected component (if applicable):
> 4.0.0
> 
> How reproducible:
> Always
> 
> Steps to Reproduce:
> Setup a cluster using docker containers as detailed here:
> https://hackmd.io/-yC3Ol68SwaRWr8bzaL8pw#
> 
> 
> Actual results:
> None of the operations failed, the logs were having the above messages, even
> after a remount on the client.

(In reply to Raghavendra G from comment #1)
> (In reply to Shyamsundar from comment #0)
> > and, this message
> > I [MSGID: 109063] [dht-layout.c:693:dht_layout_normalize] 0-patchy-dht:
> > Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> 
> This is a false positive of an error. These errors are normally seen when a
> lookup races with mkdir and finds a directory with layout _yet_ to be set.
> IOW, its a transient condition which gets resolved once either mkdir or
> lookup completes.

Note, on multiple mounts and unmounts, this did not go away.

> Note, on multiple mounts and unmounts, this did not go away.

During every mkdir this error msg can be seen (mounting/umounting has no impact). However, if you are seeing this error during lookup on an existing directory, its a cause for concern.

(In reply to Raghavendra G from comment #3)
> > Note, on multiple mounts and unmounts, this did not go away.
> 
> During every mkdir this error msg can be seen (mounting/umounting has no
> impact). However, if you are seeing this error during lookup on an existing
> directory, its a cause for concern.

Ah! apologies, I meant this for the first message which I repeat here,

> Post upgrade, client mount is seeing this error
> I [MSGID: 109005] [dht-selfheal.c:2328:dht_selfheal_directory] 0-patchy-dht:
> Directory selfheal failed: Unable to form layout for directory /

Also, the anomalies message even though a false positive, should be suppressed, as this is noise.

Further, I forgot to report one more error in the logs as follows for every mkdir, consider this as well a part of this bug report.

I [dict.c:491:dict_get] (-->/usr/lib64/glusterfs/4.0.0/xlator/cluster/distribute.so(+0x21740) [0x7fce5c1cc740] -->/usr/lib64/glusterfs/4.0.0/xlator/cluster/distribute.so(+0x42d55) [0x7fce5c1edd55] -->/lib64/libglusterfs.so.0(dict_get+0x10c) [0x7fce63d4187c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Invalid argument]

This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.