Description of problem: ----------------------- 2 node cluster,4 clients writing in their specific subdirectories (using Bonnie,dbench,kernel untar). Ganesha crashed on one of my nodes with the following BT : <BT> #0 mem_get_from_pool (pt_pool=0x7fcce0000e88) at mem-pool.c:644 #1 0x00007fcf3c0c426b in mem_get (mem_pool=mem_pool@entry=0x7fcf3c365940 <pools>) at mem-pool.c:684 #2 0x00007fcf3c0c4323 in mem_get0 (mem_pool=0x7fcf3c365940 <pools>) at mem-pool.c:581 #3 0x00007fcf3c09491c in get_new_data () at dict.c:40 #4 0x00007fcf3c0963d7 in bin_to_data (value=<optimized out>, len=<optimized out>) at dict.c:935 #5 0x00007fcf3c096476 in dict_set_bin_common (this=<optimized out>, key=<optimized out>, ptr=<optimized out>, size=<optimized out>, is_static=<optimized out>) at dict.c:2300 #6 0x00007fcf3c097efb in dict_set_static_bin (this=<optimized out>, key=<optimized out>, ptr=<optimized out>, size=<optimized out>) at dict.c:2329 #7 0x00007fcf3c38191e in glfs_resolve_component (fs=fs@entry=0x562567e93060, subvol=subvol@entry=0x7fcf2404b3e0, parent=parent@entry=0x7fcdd800ed10, component=component@entry=0x7fcce003ad00 "LWPSAV0.TMP", iatt=iatt@entry=0x7fcedb79bfb0, force_lookup=<optimized out>) at glfs-resolve.c:379 #8 0x00007fcf3c381dfb in priv_glfs_resolve_at (fs=fs@entry=0x562567e93060, subvol=subvol@entry=0x7fcf2404b3e0, at=at@entry=0x7fcdd800ed10, origpath=origpath@entry=0x7fcce001e340 "LWPSAV0.TMP", loc=loc@entry=0x7fcedb79c0b0, iatt=iatt@entry=0x7fcedb79c0f0, follow=follow@entry=0, reval=reval@entry=0) at glfs-resolve.c:501 #9 0x00007fcf3c3837b4 in pub_glfs_h_lookupat (fs=0x562567e93060, parent=<optimized out>, path=path@entry=0x7fcce001e340 "LWPSAV0.TMP", stat=stat@entry=0x7fcedb79c1d0, follow=follow@entry=0) at glfs-handleops.c:102 #10 0x00007fcf3c383898 in pub_glfs_h_lookupat34 (fs=<optimized out>, parent=<optimized out>, path=path@entry=0x7fcce001e340 "LWPSAV0.TMP", stat=stat@entry=0x7fcedb79c1d0) at glfs-handleops.c:133 #11 0x00007fcf3c7a039f in lookup (parent=0x7fcdd8014348, path=0x7fcce001e340 "LWPSAV0.TMP", handle=0x7fcedb79c310, attrs_out=0x7fcedb79c320) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/FSAL_GLUSTER/handle.c:117 #12 0x0000562566fd2f8f in mdc_lookup_uncached (mdc_parent=mdc_parent@entry=0x7fcdd8005770, name=name@entry=0x7fcce001e340 "LWPSAV0.TMP", new_entry=new_entry@entry=0x7fcedb79c4c0, attrs_out=attrs_out@entry=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1046 #13 0x0000562566fd5c5b in mdc_lookup (mdc_parent=0x7fcdd8005770, name=0x7fcce001e340 "LWPSAV0.TMP", uncached=uncached@entry=true, new_entry=new_entry@entry=0x7fcedb79c4c0, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1004 #14 0x0000562566fcab7b in mdcache_lookup (parent=<optimized out>, name=<optimized out>, handle=0x7fcedb79c578, attrs_out=<optimized out>) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:167 #15 0x0000562566f03ac7 in fsal_lookup (parent=parent@entry=0x7fcdd80057a8, name=0x7fcce001e340 "LWPSAV0.TMP", obj=obj@entry=0x7fcedb79c578, attrs_out=attrs_out@entry=0x0) at /usr/src/debug/nfs-ganesha-2.4.4/src/FSAL/fsal_helper.c:712 #16 0x0000562566ef0ce6 in open4_ex (arg=arg@entry=0x7fcd3000b2f8, data=data@entry=0x7fcedb79d180, res_OPEN4=res_OPEN4@entry=0x7fcce0014198, clientid=<optimized out>, owner=0x7fcdd0001210, file_state=file_state@entry=0x7fcedb79cfa0, new_state=new_state@entry=0x7fcedb79cf8f) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_open.c:1257 #17 0x0000562566f39469 in nfs4_op_open (op=0x7fcd3000b2f0, data=0x7fcedb79d180, resp=0x7fcce0014190) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_op_open.c:1845 #18 0x0000562566f2b97d in nfs4_Compound (arg=<optimized out>, req=<optimized out>, res=0x7fcce0020be0) at /usr/src/debug/nfs-ganesha-2.4.4/src/Protocols/NFS/nfs4_Compound.c:734 #19 0x0000562566f1cb1c in nfs_rpc_execute (reqdata=reqdata@entry=0x7fcd30012a60) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1281 #20 0x0000562566f1e18a in worker_run (ctx=0x56256af215e0) at /usr/src/debug/nfs-ganesha-2.4.4/src/MainNFSD/nfs_worker_thread.c:1548 #21 0x0000562566fa7889 in fridgethr_start_routine (arg=0x56256af215e0) at /usr/src/debug/nfs-ganesha-2.4.4/src/support/fridgethr.c:550 #22 0x00007fcf3f704e25 in start_thread (arg=0x7fcedb79e700) at pthread_create.c:308 #23 0x00007fcf3edd234d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 (gdb) </BT> Version-Release number of selected component (if applicable): ------------------------------------------------------------- nfs-ganesha-debuginfo-2.4.4-10.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-31.el7rhgs.x86_64 How reproducible: ----------------- Reporting the first occurrence of the crash. Additional info: --------------- [root@gqas007 tmp]# gluster v info Volume Name: butcher Type: Distributed-Disperse Volume ID: 22c652d8-0754-438a-8131-373bad7c12ab Status: Started Snapshot Count: 0 Number of Bricks: 4 x (4 + 2) = 24 Transport-type: tcp Bricks: Brick1: gqas014.sbu.lab.eng.bos.redhat.com:/bricks1/A1 Brick2: gqas007.sbu.lab.eng.bos.redhat.com:/bricks1/A1 Brick3: gqas014.sbu.lab.eng.bos.redhat.com:/bricks2/A1 Brick4: gqas007.sbu.lab.eng.bos.redhat.com:/bricks2/A1 Brick5: gqas014.sbu.lab.eng.bos.redhat.com:/bricks3/A1 Brick6: gqas007.sbu.lab.eng.bos.redhat.com:/bricks3/A1 Brick7: gqas014.sbu.lab.eng.bos.redhat.com:/bricks4/A1 Brick8: gqas007.sbu.lab.eng.bos.redhat.com:/bricks4/A1 Brick9: gqas014.sbu.lab.eng.bos.redhat.com:/bricks5/A1 Brick10: gqas007.sbu.lab.eng.bos.redhat.com:/bricks5/A1 Brick11: gqas014.sbu.lab.eng.bos.redhat.com:/bricks6/A1 Brick12: gqas007.sbu.lab.eng.bos.redhat.com:/bricks6/A1 Brick13: gqas014.sbu.lab.eng.bos.redhat.com:/bricks7/A1 Brick14: gqas007.sbu.lab.eng.bos.redhat.com:/bricks7/A1 Brick15: gqas014.sbu.lab.eng.bos.redhat.com:/bricks8/A1 Brick16: gqas007.sbu.lab.eng.bos.redhat.com:/bricks8/A1 Brick17: gqas014.sbu.lab.eng.bos.redhat.com:/bricks9/A1 Brick18: gqas007.sbu.lab.eng.bos.redhat.com:/bricks9/A1 Brick19: gqas014.sbu.lab.eng.bos.redhat.com:/bricks10/A1 Brick20: gqas007.sbu.lab.eng.bos.redhat.com:/bricks10/A1 Brick21: gqas014.sbu.lab.eng.bos.redhat.com:/bricks11/A1 Brick22: gqas007.sbu.lab.eng.bos.redhat.com:/bricks11/A1 Brick23: gqas014.sbu.lab.eng.bos.redhat.com:/bricks12/A1 Brick24: gqas007.sbu.lab.eng.bos.redhat.com:/bricks12/A1 Options Reconfigured: ganesha.enable: on network.inode-lru-limit: 50000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on transport.address-family: inet nfs.disable: on nfs-ganesha: enable cluster.enable-shared-storage: enable