Description of problem:
-----------------------

This appears to come from DHT/WB,but raising a Ganesha bug for initial RCA.

4 node setup,4 clients mount the gluster volume(2 clients:1 server).

Workload Details : SmallFile mkdirs,bonnie++

Mount Type : v4

Ganesha crashed on one of my nodes and dumped the following core :

<BT>

(gdb) bt
#0  dht_layout_get (this=0x7fe9e0012e50, inode=inode@entry=0x7fe69c0b7840) at dht-layout.c:77
#1  0x00007fe9ecda756e in dht_local_init (frame=frame@entry=0x7fe7a404dd60, loc=loc@entry=0x7fe97d7a7b40, fd=fd@entry=0x0, fop=fop@entry=GF_FOP_STAT) at dht-helper.c:670
#2  0x00007fe9ece0592c in dht_stat (frame=0x7fe7a404dd60, this=0x7fe9e0012e50, loc=0x7fe97d7a7b40, xdata=0x0) at dht-inode-read.c:319
#3  0x00007fe9ecb95809 in wb_stat (frame=frame@entry=0x7fe7a4018600, this=this@entry=0x7fe9e00147e0, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at write-behind.c:1903
#4  0x00007fe9f9d9334e in default_stat (frame=frame@entry=0x7fe7a4018600, this=this@entry=0x7fe9e0016230, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at defaults.c:2647
#5  0x00007fe9f9d9334e in default_stat (frame=frame@entry=0x7fe7a4018600, this=this@entry=0x7fe9e0017790, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at defaults.c:2647
#6  0x00007fe9f9d9334e in default_stat (frame=frame@entry=0x7fe7a4018600, this=this@entry=0x7fe9e00192c0, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at defaults.c:2647
#7  0x00007fe9f9d9334e in default_stat (frame=frame@entry=0x7fe7a4018600, this=this@entry=0x7fe9e001ab60, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at defaults.c:2647
#8  0x00007fe9f9d9334e in default_stat (frame=0x7fe7a4018600, this=<optimized out>, loc=0x7fe97d7a7b40, xdata=0x0) at defaults.c:2647
#9  0x00007fe9e7bc2f71 in io_stats_stat (frame=frame@entry=0x7fe7a4052360, this=this@entry=0x7fe9e001d760, loc=loc@entry=0x7fe97d7a7b40, xdata=xdata@entry=0x0) at io-stats.c:2719
#10 0x00007fe9f9d9334e in default_stat (frame=0x7fe7a4052360, this=<optimized out>, loc=0x7fe97d7a7b40, xdata=0x0) at defaults.c:2647
#11 0x00007fe9e79ad21e in meta_stat (frame=0x7fe7a4052360, this=0x7fe9e001f1b0, loc=0x7fe97d7a7b40, xdata=0x0) at meta.c:92
#12 0x00007fe9f9d5dfef in syncop_stat (subvol=subvol@entry=0x7fe9e001f1b0, loc=loc@entry=0x7fe97d7a7b40, stbuf=stbuf@entry=0x7fe97d7a7b80, xdata_in=xdata_in@entry=0x0, xdata_out=xdata_out@entry=0x0)
    at syncop.c:2425
#13 0x00007fe9f9fffbd3 in pub_glfs_h_stat (fs=0x5648f612daf0, object=<optimized out>, stat=stat@entry=0x7fe97d7a7c48) at glfs-handleops.c:233
#14 0x00007fe9fa41b7e8 in getattrs (obj_hdl=0x7fe69c140778, attrs=0x7fe97d7a8d40) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/FSAL_GLUSTER/handle.c:770
#15 0x00005648f4c85de4 in mdcache_refresh_attrs (entry=entry@entry=0x7fe69c127720, need_acl=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:967
#16 0x00005648f4c868ea in mdcache_getattrs (obj_hdl=0x7fe69c127758, attrs_out=0x7fe97d7a8fd0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1060
#17 0x00005648f4c0b877 in file_To_Fattr (data=data@entry=0x7fe97d7a9180, request_mask=1433550, attr=attr@entry=0x7fe97d7a8fd0, Fattr=Fattr@entry=0x7fe7a41acbe0, Bitmap=Bitmap@entry=0x7fe7bc0a1218)
    at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs_proto_tools.c:3312
#18 0x00005648f4be8b02 in nfs4_op_getattr (op=0x7fe7bc0a1210, data=0x7fe97d7a9180, resp=0x7fe7a41acbd0) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_getattr.c:140
#19 0x00005648f4be397d in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7fe7a4097d30) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_Compound.c:734
#20 0x00005648f4bd4b1c in nfs_rpc_execute (reqdata=reqdata@entry=0x7fe7bc0896d0) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1281
#21 0x00005648f4bd618a in worker_run (ctx=0x5648f9223910) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1548
#22 0x00005648f4c5f889 in fridgethr_start_routine (arg=0x5648f9223910) at /usr/src/debug/nfs-ganesha-2.4.4/src/support/fridgethr.c:550
#23 0x00007fe9fd380e25 in start_thread () from /lib64/libpthread.so.0
#24 0x00007fe9fca4e34d in clone () from /lib64/libc.so.6
(gdb) 

</BT>

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

nfs-ganesha-2.4.4-10.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-32.el7rhgs.x86_64


How reproducible:
-----------------

Reporting the first occurrence.

Additional info:
----------------

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 6ade5657-45e2-43c7-8098-774417789a5e
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
ganesha.enable: on
features.cache-invalidation: on
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable