Bug 1403666 - [Perf] : Ganesha crash in afr_inodelk_init while running IO from multiple clients
Summary: [Perf] : Ganesha crash in afr_inodelk_init while running IO from multiple cli...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: nfs-ganesha
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: RHGS 3.2.0
Assignee: Daniel Gryniewicz
QA Contact: Ambarish
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1394717
TreeView+ depends on / blocked
 
Reported: 2016-12-12 06:51 UTC by Ambarish
Modified: 2017-03-28 06:54 UTC (History)
15 users (show)

Fixed In Version: nfs-ganesha-2.4.1-4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1394717
Environment:
Last Closed: 2017-03-23 06:26:29 UTC
Embargoed:

Attachments (Terms of Use)
Potential fix? (6.80 KB, text/plain)
2016-12-15 19:10 UTC, Daniel Gryniewicz 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:0493 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.2.0 nfs-ganesha bug fix and enhancement update 2017-03-23 09:19:13 UTC

Description Ambarish 2016-12-12 06:51:51 UTC 
This issue was originally reported as part of https://bugzilla.redhat.com/show_bug.cgi?id=1394717. 
Creating a new BZ post discussion with Ganesha guys in replicate layer : 

(gdb) bt
#0  0x00007f14578cbf59 in _int_malloc () from /lib64/libc.so.6
#1  0x00007f14578cea14 in calloc () from /lib64/libc.so.6
#2  0x00007f13cc6b85e8 in __gf_calloc (nmemb=nmemb@entry=1, size=<optimized out>, type=type@entry=155, 
    typestr=typestr@entry=0x7f13ba0bffeb "gf_afr_mt_char") at mem-pool.c:117
#3  0x00007f13ba0b3083 in afr_inodelk_init (lk=lk@entry=0x7f13ab2094d8, dom=<optimized out>, 
    child_count=<optimized out>) at afr-common.c:4718
#4  0x00007f13ba0b3273 in afr_transaction_local_init (local=local@entry=0x7f13ab208e80, 
    this=this@entry=0x7f13ac00c550) at afr-common.c:4742
#5  0x00007f13ba08b8d4 in afr_transaction (frame=frame@entry=0x7f13c33ed77c, this=this@entry=0x7f13ac00c550, 
    type=type@entry=AFR_DATA_TRANSACTION) at afr-transaction.c:2574
#6  0x00007f13ba07b4cb in afr_do_writev (frame=frame@entry=0x7f13c33d1e18, this=this@entry=0x7f13ac00c550)
    at afr-inode-write.c:489
#7  0x00007f13ba07be44 in afr_writev (frame=0x7f13c33d1e18, this=0x7f13ac00c550, fd=0x7f13a846f1dc, 
    vector=0x7f109e1fddb0, count=1, offset=3656175615, flags=0, iobref=0x7f11ec08a720, xdata=0x0)
    at afr-inode-write.c:559
#8  0x00007f13b9e30639 in dht_writev (frame=<optimized out>, this=<optimized out>, fd=0x7f13a846f1dc, 
    vector=0x7f109e1fddb0, count=1, off=3656175615, flags=0, iobref=0x7f11ec08a720, xdata=0x0)
    at dht-inode-write.c:192
#9  0x00007f13b9bcb0d0 in wb_fulfill_head (wb_inode=wb_inode@entry=0x7f13a0cf4050, head=0x7f11f0100600)
    at write-behind.c:1049
#10 0x00007f13b9bcb2cb in wb_fulfill (wb_inode=wb_inode@entry=0x7f13a0cf4050, 
    liabilities=liabilities@entry=0x7f109e1fdf10) at write-behind.c:1130
#11 0x00007f13b9bcc046 in wb_process_queue (wb_inode=wb_inode@entry=0x7f13a0cf4050) at write-behind.c:1550
#12 0x00007f13b9bcc734 in wb_writev (frame=0x7f13c347650c, this=<optimized out>, fd=<optimized out>, 
    vector=0x7f121c0444e0, count=1, offset=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0)
    at write-behind.c:1657
#13 0x00007f13b99bceeb in ra_writev (frame=0x7f13c33f166c, this=0x7f13ac011970, fd=0x7f13a846f1dc, 
    vector=0x7f121c0444e0, count=1, offset=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0)
    at read-ahead.c:684
#14 0x00007f13cc70bcf5 in default_writev (frame=0x7f13c33f166c, this=0x7f13ac012dc0, fd=0x7f13a846f1dc, 
    vector=0x7f121c0444e0, count=1, off=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0) at defaults.c:2543
---Type <return> to continue, or q <return> to quit---
#15 0x00007f13b95a2764 in ioc_writev (frame=0x7f13c33cf308, this=0x7f13ac014320, fd=0x7f13a846f1dc, 
    vector=0x7f121c0444e0, count=1, offset=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0) at io-cache.c:1263
#16 0x00007f13b9397189 in qr_writev (frame=0x7f13c33f00e4, this=0x7f13ac015a40, fd=0x7f13a846f1dc, 
    iov=0x7f121c0444e0, count=1, offset=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0) at quick-read.c:636
#17 0x00007f13cc7257ec in default_writev_resume (frame=0x7f13c3417b3c, this=0x7f13ac016f00, fd=0x7f13a846f1dc, 
    vector=0x7f121c0444e0, count=1, off=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0) at defaults.c:1849
#18 0x00007f13cc6b50d8 in call_resume_wind (stub=0x7f13c2e29a94) at call-stub.c:2045
#19 0x00007f13cc6b556d in call_resume (stub=0x7f13c2e29a94) at call-stub.c:2508
#20 0x00007f13b918d2e8 in open_and_resume (this=this@entry=0x7f13ac016f00, fd=fd@entry=0x7f13a846f1dc, 
    stub=0x7f13c2e29a94) at open-behind.c:245
#21 0x00007f13b918d350 in ob_writev (frame=0x7f13c3417b3c, this=0x7f13ac016f00, fd=0x7f13a846f1dc, 
    iov=<optimized out>, count=<optimized out>, offset=<optimized out>, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0)
    at open-behind.c:423
#22 0x00007f13cc7257ec in default_writev_resume (frame=0x7f13c345a28c, this=0x7f13ac0183c0, fd=0x7f13a846f1dc, 
    vector=0x7f10ec05a910, count=1, off=3656167423, flags=0, iobref=0x7f10ec02d4d0, xdata=0x0) at defaults.c:1849
#23 0x00007f13cc6b50d8 in call_resume_wind (stub=0x7f13c2e52de4) at call-stub.c:2045
#24 0x00007f13cc6b556d in call_resume (stub=0x7f13c2e52de4) at call-stub.c:2508
#25 0x00007f13b8f83857 in iot_worker (data=0x7f13ac027840) at io-threads.c:220
#26 0x00007f1458276dc5 in start_thread () from /lib64/libpthread.so.0
#27 0x00007f145794573d in clone () from /lib64/libc.so.6
(gdb) 


sos ,ganesha logs,core for the reproducer :


http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1394702/28112016/

More details in https://bugzilla.redhat.com/show_bug.cgi?id=1394717
Comment 3 Ravishankar N 2016-12-14 10:43:17 UTC 
Hi Ambarish,
Since the machines were re-purposed since the last tests, I have again installed downstream gluster from source (compiled with asan) on all 4 servers:gqas{005,006,011,013}.sbu.lab.eng.bos.redhat.com. 

As discussed, the source was compiled with the HEAD retained at the following commit:
2ca340c cluster/dht: A hard link is lost during rebalance + lookup

Let us not release these machines until we complete the RCA for the crashes.

Soumya/Jiffin,
Please update once you have compiled and installed ganesha with asan on these machines, then Ambarish can resume testing.
Comment 4 Jiffin 2016-12-15 04:15:03 UTC 
(In reply to Ravishankar N from comment #3)
> Hi Ambarish,
> Since the machines were re-purposed since the last tests, I have again
> installed downstream gluster from source (compiled with asan) on all 4
> servers:gqas{005,006,011,013}.sbu.lab.eng.bos.redhat.com. 
> 
> As discussed, the source was compiled with the HEAD retained at the
> following commit:
> 2ca340c cluster/dht: A hard link is lost during rebalance + lookup
> 
> Let us not release these machines until we complete the RCA for the crashes.
> 
> Soumya/Jiffin,
> Please update once you have compiled and installed ganesha with asan on
> these machines, then Ambarish can resume testing.

I have complied source downstream ganesha bits with asan which QA is currently testing
Comment 5 Jiffin 2016-12-15 04:15:49 UTC 
(In reply to Ravishankar N from comment #3)
> Hi Ambarish,
> Since the machines were re-purposed since the last tests, I have again
> installed downstream gluster from source (compiled with asan) on all 4
> servers:gqas{005,006,011,013}.sbu.lab.eng.bos.redhat.com. 
> 
> As discussed, the source was compiled with the HEAD retained at the
> following commit:
> 2ca340c cluster/dht: A hard link is lost during rebalance + lookup
> 
> Let us not release these machines until we complete the RCA for the crashes.
> 
> Soumya/Jiffin,
> Please update once you have compiled and installed ganesha with asan on
> these machines, then Ambarish can resume testing.

I have completed compilation of source downstream ganesha bits with asan which QA is currently testing on those machines
Comment 6 Ambarish 2016-12-15 16:04:07 UTC 
We do have a reproducer post compiling Ganesha and Gluster bits with ASAN,while running dd and untar from multiple heterogeneous mounts.

Ganesha did not dump core,but I believe it's the same memory corruption issue that has been reported in multiple BZs.

Copying output from stdout(running Ganesha in foreground):


************
On gqas013
************

=================================================================
==15962== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a008d5a40 at pc 0x41ee67 bp 0x7f940473de30 sp 0x7f940473de20
WRITE of size 8 at 0x604a008d5a40 thread T52
    #0 0x41ee66 (/usr/bin/ganesha.nfsd+0x41ee66)
    #1 0x41ef11 (/usr/bin/ganesha.nfsd+0x41ef11)
    #2 0x41ffc0 (/usr/bin/ganesha.nfsd+0x41ffc0)
    #3 0x68dc7e (/usr/bin/ganesha.nfsd+0x68dc7e)
    #4 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
    #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
    #7 0x606683 (/usr/bin/ganesha.nfsd+0x606683)
    #8 0x52e904 (/usr/bin/ganesha.nfsd+0x52e904)
    #9 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #10 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #11 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #12 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
    #13 0x7f9439747dc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
    #14 0x7f943757073c (/usr/lib64/libc-2.17.so+0xf773c)
0x604a008d5a40 is located 64 bytes inside of 1480-byte region [0x604a008d5a00,0x604a008d5fc8)
freed by thread T208 here:
    #0 0x7f9439972009 (/usr/lib64/libasan.so.0.0.0+0x16009)
    #1 0x66a342 (/usr/bin/ganesha.nfsd+0x66a342)
    #2 0x66a3c1 (/usr/bin/ganesha.nfsd+0x66a3c1)
    #3 0x67497e (/usr/bin/ganesha.nfsd+0x67497e)
    #4 0x690fd5 (/usr/bin/ganesha.nfsd+0x690fd5)
    #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
    #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
    #7 0x606683 (/usr/bin/ganesha.nfsd+0x606683)
    #8 0x52e904 (/usr/bin/ganesha.nfsd+0x52e904)
    #9 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #10 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #11 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #12 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
previously allocated by thread T278 here:
    #0 0x7f9439972225 (/usr/lib64/libasan.so.0.0.0+0x16225)
    #1 0x66a2f7 (/usr/bin/ganesha.nfsd+0x66a2f7)
    #2 0x66a3a3 (/usr/bin/ganesha.nfsd+0x66a3a3)
    #3 0x67208b (/usr/bin/ganesha.nfsd+0x67208b)
    #4 0x672339 (/usr/bin/ganesha.nfsd+0x672339)
    #5 0x68daaa (/usr/bin/ganesha.nfsd+0x68daaa)
    #6 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #7 0x67a24a (/usr/bin/ganesha.nfsd+0x67a24a)
    #8 0x693a77 (/usr/bin/ganesha.nfsd+0x693a77)
    #9 0x69345d (/usr/bin/ganesha.nfsd+0x69345d)
    #10 0x67a739 (/usr/bin/ganesha.nfsd+0x67a739)
    #11 0x448560 (/usr/bin/ganesha.nfsd+0x448560)
    #12 0x520b93 (/usr/bin/ganesha.nfsd+0x520b93)
    #13 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #14 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #15 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #16 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
Thread T52 created by T0 here:
    #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T208 created by T0 here:
    #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T278 created by T0 here:
    #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
Shadow bytes around the buggy address:
  0x0c09c0112af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0112b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0112b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0112b20: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c09c0112b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c0112b40: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c0112b50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0112b60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0112b70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0112b80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0112b90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05



**********
On gqas005
**********

==2572== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a0221ed40 at pc 0x41ee67 bp 0x7fbbd4d99d60 sp 0x7fbbd4d99d50
WRITE of size 8 at 0x604a0221ed40 thread T142
    #0 0x41ee66 (/usr/bin/ganesha.nfsd+0x41ee66)
    #1 0x41ef11 (/usr/bin/ganesha.nfsd+0x41ef11)
    #2 0x41ffc0 (/usr/bin/ganesha.nfsd+0x41ffc0)
    #3 0x68dc7e (/usr/bin/ganesha.nfsd+0x68dc7e)
    #4 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
    #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
    #7 0x4ddaa1 (/usr/bin/ganesha.nfsd+0x4ddaa1)
    #8 0x4de093 (/usr/bin/ganesha.nfsd+0x4de093)
    #9 0x4a2147 (/usr/bin/ganesha.nfsd+0x4a2147)
    #10 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #11 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #12 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #13 0x7fbc480a1a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
    #14 0x7fbc47e73dc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
    #15 0x7fbc45c9c73c (/usr/lib64/libc-2.17.so+0xf773c)
0x604a0221ed40 is located 64 bytes inside of 1480-byte region [0x604a0221ed00,0x604a0221f2c8)
freed by thread T254 here:
    #0 0x7fbc4809e009 (/usr/lib64/libasan.so.0.0.0+0x16009)
    #1 0x66a342 (/usr/bin/ganesha.nfsd+0x66a342)
    #2 0x66a3c1 (/usr/bin/ganesha.nfsd+0x66a3c1)
    #3 0x67497e (/usr/bin/ganesha.nfsd+0x67497e)
    #4 0x690fd5 (/usr/bin/ganesha.nfsd+0x690fd5)
    #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
    #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
    #7 0x4ddaa1 (/usr/bin/ganesha.nfsd+0x4ddaa1)
    #8 0x4de093 (/usr/bin/ganesha.nfsd+0x4de093)
    #9 0x4a2147 (/usr/bin/ganesha.nfsd+0x4a2147)
    #10 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #11 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #12 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #13 0x7fbc480a1a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
previously allocated by thread T44 here:
    #0 0x7fbc4809e225 (/usr/lib64/libasan.so.0.0.0+0x16225)
    #1 0x66a2f7 (/usr/bin/ganesha.nfsd+0x66a2f7)
    #2 0x66a3a3 (/usr/bin/ganesha.nfsd+0x66a3a3)
    #3 0x67208b (/usr/bin/ganesha.nfsd+0x67208b)
    #4 0x672339 (/usr/bin/ganesha.nfsd+0x672339)
    #5 0x68daaa (/usr/bin/ganesha.nfsd+0x68daaa)
    #6 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #7 0x67a24a (/usr/bin/ganesha.nfsd+0x67a24a)
    #8 0x687d2d (/usr/bin/ganesha.nfsd+0x687d2d)
    #9 0x446cc5 (/usr/bin/ganesha.nfsd+0x446cc5)
    #10 0x44f49f (/usr/bin/ganesha.nfsd+0x44f49f)
    #11 0x4d326c (/usr/bin/ganesha.nfsd+0x4d326c)
    #12 0x4d67fb (/usr/bin/ganesha.nfsd+0x4d67fb)
    #13 0x4a2147 (/usr/bin/ganesha.nfsd+0x4a2147)
    #14 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #15 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #16 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #17 0x7fbc480a1a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
Thread T142 created by T0 here:
    #0 0x7fbc48092c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7fbc45bc6b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T254 created by T0 here:
    #0 0x7fbc48092c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7fbc45bc6b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T44 created by T0 here:
    #0 0x7fbc48092c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7fbc45bc6b34 (/usr/lib64/libc-2.17.so+0x21b34)
Shadow bytes around the buggy address:
  0x0c09c043bd50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c043bd60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c043bd70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c043bd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c043bd90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c043bda0: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c043bdb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c043bdc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c043bdd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c043bde0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c043bdf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd



***********
On gqas011
***********

==2227== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a00c160c0 at pc 0x41ee67 bp 0x7f1e359fed30 sp 0x7f1e359fed20
WRITE of size 8 at 0x604a00c160c0 thread T229
    #0 0x41ee66 (/usr/bin/ganesha.nfsd+0x41ee66)
    #1 0x41ef11 (/usr/bin/ganesha.nfsd+0x41ef11)
    #2 0x41ffc0 (/usr/bin/ganesha.nfsd+0x41ffc0)
    #3 0x68dc7e (/usr/bin/ganesha.nfsd+0x68dc7e)
    #4 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #5 0x67a24a (/usr/bin/ganesha.nfsd+0x67a24a)
    #6 0x67b6ec (/usr/bin/ganesha.nfsd+0x67b6ec)
    #7 0x44919c (/usr/bin/ganesha.nfsd+0x44919c)
    #8 0x521ac1 (/usr/bin/ganesha.nfsd+0x521ac1)
    #9 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #10 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #11 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #12 0x7f1ee4cbea97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
    #13 0x7f1ee4a90dc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
    #14 0x7f1ee28b973c (/usr/lib64/libc-2.17.so+0xf773c)
0x604a00c160c0 is located 64 bytes inside of 1480-byte region [0x604a00c16080,0x604a00c16648)
freed by thread T209 here:
    #0 0x7f1ee4cbb009 (/usr/lib64/libasan.so.0.0.0+0x16009)
    #1 0x66a342 (/usr/bin/ganesha.nfsd+0x66a342)
    #2 0x66a3c1 (/usr/bin/ganesha.nfsd+0x66a3c1)
    #3 0x67497e (/usr/bin/ganesha.nfsd+0x67497e)
    #4 0x690fd5 (/usr/bin/ganesha.nfsd+0x690fd5)
    #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
    #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
    #7 0x4ddaa1 (/usr/bin/ganesha.nfsd+0x4ddaa1)
    #8 0x4de093 (/usr/bin/ganesha.nfsd+0x4de093)
    #9 0x4a2147 (/usr/bin/ganesha.nfsd+0x4a2147)
    #10 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #11 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #12 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #13 0x7f1ee4cbea97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
previously allocated by thread T270 here:
    #0 0x7f1ee4cbb225 (/usr/lib64/libasan.so.0.0.0+0x16225)
    #1 0x66a2f7 (/usr/bin/ganesha.nfsd+0x66a2f7)
    #2 0x66a3a3 (/usr/bin/ganesha.nfsd+0x66a3a3)
    #3 0x67208b (/usr/bin/ganesha.nfsd+0x67208b)
    #4 0x672339 (/usr/bin/ganesha.nfsd+0x672339)
    #5 0x68daaa (/usr/bin/ganesha.nfsd+0x68daaa)
    #6 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
    #7 0x67a24a (/usr/bin/ganesha.nfsd+0x67a24a)
    #8 0x687d2d (/usr/bin/ganesha.nfsd+0x687d2d)
    #9 0x446cc5 (/usr/bin/ganesha.nfsd+0x446cc5)
    #10 0x44f49f (/usr/bin/ganesha.nfsd+0x44f49f)
    #11 0x51d024 (/usr/bin/ganesha.nfsd+0x51d024)
    #12 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
    #13 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
    #14 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
    #15 0x7f1ee4cbea97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
Thread T229 created by T0 here:
    #0 0x7f1ee4cafc3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f1ee27e3b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T209 created by T0 here:
    #0 0x7f1ee4cafc3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f1ee27e3b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T270 created by T0 here:
    #0 0x7f1ee4cafc3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
    #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
    #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
    #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
    #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
    #6 0x7f1ee27e3b34 (/usr/lib64/libc-2.17.so+0x21b34)
Shadow bytes around the buggy address:
  0x0c09c017abc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c017abd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c017abe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c017abf0: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c09c017ac00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c017ac10: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c017ac20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c017ac30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c017ac40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c017ac50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c017ac60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Comment 7 Soumya Koduri 2016-12-15 16:37:38 UTC 
(In reply to Ambarish from comment #6)
> We do have a reproducer post compiling Ganesha and Gluster bits with
> ASAN,while running dd and untar from multiple heterogeneous mounts.
> 
> Ganesha did not dump core,but I believe it's the same memory corruption
> issue that has been reported in multiple BZs.
> 
> Copying output from stdout(running Ganesha in foreground):
> 
> 
> ************
> On gqas013
> ************
> 
> =================================================================
> ==15962== ERROR: AddressSanitizer: heap-use-after-free on address
> 0x604a008d5a40 at pc 0x41ee67 bp 0x7f940473de30 sp 0x7f940473de20
> WRITE of size 8 at 0x604a008d5a40 thread T52
>     #0 0x41ee66 (/usr/bin/ganesha.nfsd+0x41ee66)
>     #1 0x41ef11 (/usr/bin/ganesha.nfsd+0x41ef11)
>     #2 0x41ffc0 (/usr/bin/ganesha.nfsd+0x41ffc0)
>     #3 0x68dc7e (/usr/bin/ganesha.nfsd+0x68dc7e)
>     #4 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
>     #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
>     #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
>     #7 0x606683 (/usr/bin/ganesha.nfsd+0x606683)
>     #8 0x52e904 (/usr/bin/ganesha.nfsd+0x52e904)
>     #9 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
>     #10 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
>     #11 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
>     #12 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
>     #13 0x7f9439747dc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
>     #14 0x7f943757073c (/usr/lib64/libc-2.17.so+0xf773c)
> 0x604a008d5a40 is located 64 bytes inside of 1480-byte region
> [0x604a008d5a00,0x604a008d5fc8)
> freed by thread T208 here:
>     #0 0x7f9439972009 (/usr/lib64/libasan.so.0.0.0+0x16009)
>     #1 0x66a342 (/usr/bin/ganesha.nfsd+0x66a342)
>     #2 0x66a3c1 (/usr/bin/ganesha.nfsd+0x66a3c1)
>     #3 0x67497e (/usr/bin/ganesha.nfsd+0x67497e)
>     #4 0x690fd5 (/usr/bin/ganesha.nfsd+0x690fd5)
>     #5 0x691caa (/usr/bin/ganesha.nfsd+0x691caa)
>     #6 0x684851 (/usr/bin/ganesha.nfsd+0x684851)
>     #7 0x606683 (/usr/bin/ganesha.nfsd+0x606683)
>     #8 0x52e904 (/usr/bin/ganesha.nfsd+0x52e904)
>     #9 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
>     #10 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
>     #11 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
>     #12 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
> previously allocated by thread T278 here:
>     #0 0x7f9439972225 (/usr/lib64/libasan.so.0.0.0+0x16225)
>     #1 0x66a2f7 (/usr/bin/ganesha.nfsd+0x66a2f7)
>     #2 0x66a3a3 (/usr/bin/ganesha.nfsd+0x66a3a3)
>     #3 0x67208b (/usr/bin/ganesha.nfsd+0x67208b)
>     #4 0x672339 (/usr/bin/ganesha.nfsd+0x672339)
>     #5 0x68daaa (/usr/bin/ganesha.nfsd+0x68daaa)
>     #6 0x690023 (/usr/bin/ganesha.nfsd+0x690023)
>     #7 0x67a24a (/usr/bin/ganesha.nfsd+0x67a24a)
>     #8 0x693a77 (/usr/bin/ganesha.nfsd+0x693a77)
>     #9 0x69345d (/usr/bin/ganesha.nfsd+0x69345d)
>     #10 0x67a739 (/usr/bin/ganesha.nfsd+0x67a739)
>     #11 0x448560 (/usr/bin/ganesha.nfsd+0x448560)
>     #12 0x520b93 (/usr/bin/ganesha.nfsd+0x520b93)
>     #13 0x47a255 (/usr/bin/ganesha.nfsd+0x47a255)
>     #14 0x47b9d3 (/usr/bin/ganesha.nfsd+0x47b9d3)
>     #15 0x62541e (/usr/bin/ganesha.nfsd+0x62541e)
>     #16 0x7f9439975a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
> Thread T52 created by T0 here:
>     #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
>     #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
>     #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
>     #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
>     #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
>     #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
>     #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
> Thread T208 created by T0 here:
>     #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
>     #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
>     #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
>     #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
>     #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
>     #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
>     #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
> Thread T278 created by T0 here:
>     #0 0x7f9439966c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
>     #1 0x62d525 (/usr/bin/ganesha.nfsd+0x62d525)
>     #2 0x47bf46 (/usr/bin/ganesha.nfsd+0x47bf46)
>     #3 0x48d94e (/usr/bin/ganesha.nfsd+0x48d94e)
>     #4 0x48fbc4 (/usr/bin/ganesha.nfsd+0x48fbc4)
>     #5 0x41d81d (/usr/bin/ganesha.nfsd+0x41d81d)
>     #6 0x7f943749ab34 (/usr/lib64/libc-2.17.so+0x21b34)
> Shadow bytes around the buggy address:
>   0x0c09c0112af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c09c0112b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c09c0112b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>   0x0c09c0112b20: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
>   0x0c09c0112b30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
> =>0x0c09c0112b40: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
>   0x0c09c0112b50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0c09c0112b60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0c09c0112b70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0c09c0112b80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
>   0x0c09c0112b90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
> Shadow byte legend (one shadow byte represents 8 application bytes):
>   Addressable:           00
>   Partially addressable: 01 02 03 04 05
> 


==================
> WRITE of size 8 at 0x604a008d5a40 thread T52

(gdb) list *0x41ee66
0x41ee66 is in __glist_add (/root/ravi/nfs-ganesha/src/include/gsh_list.h:79).
74				       struct glist_head *right, struct glist_head *elt)
75	{
76		elt->prev = left;
77		elt->next = right;
78		left->next = elt;
79		right->prev = elt;
80	}
81	
82	static inline void glist_add_tail(struct glist_head *head,
83					  struct glist_head *elt)
(gdb) list *0x41ef11
0x41ef11 is in glist_add (/root/ravi/nfs-ganesha/src/include/gsh_list.h:92).
87	}
88	
89	/* add after the specified entry*/
90	static inline void glist_add(struct glist_head *head, struct glist_head *elt)
91	{
92		__glist_add(head, head->next, elt);
93	}
94	
95	static inline void glist_del(struct glist_head *node)
96	{
(gdb) list *0x41ffc0
0x41ffc0 is in fsal_obj_handle_init (/root/ravi/nfs-ganesha/src/FSAL/commonlib.c:183).
178			PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP);
179	#endif
180		PTHREAD_RWLOCK_init(&obj->lock, &attrs);
181	
182		PTHREAD_RWLOCK_wrlock(&obj->fsal->lock);
183		glist_add(&obj->fsal->handles, &obj->handles);
184		PTHREAD_RWLOCK_unlock(&obj->fsal->lock);
185	}
186	
187	void fsal_obj_handle_fini(struct fsal_obj_handle *obj)
(gdb) list *0x68dc7e
0x68dc7e is in mdcache_alloc_handle (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:129).
124		result->obj_handle.fsid = sub_handle->fsid;
125		result->obj_handle.fileid = sub_handle->fileid;
126		result->obj_handle.fs = fs;
127	
128		/* default handlers */
129		fsal_obj_handle_init(&result->obj_handle, &export->export,
130				     sub_handle->type);
131		/* mdcache handlers */
132		mdcache_handle_ops_init(&result->obj_handle.obj_ops);
133		/* state */
(gdb) list *0x690023
0x690023 is in mdcache_new_entry (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411).
406		}
407	
408		/* !LATCHED */
409	
410		/* We did not find the object.  Pull an entry off the LRU. */
411		nentry = mdcache_alloc_handle(export, sub_handle, sub_handle->fs);
412		if (!nentry) {
413			LogCrit(COMPONENT_CACHE_INODE, "mdcache_alloc_handle failed");
414			status = fsalstat(ERR_FSAL_NOMEM, 0);
415			goto out_release;
(gdb) list *0x691caa
0x691caa is in mdcache_locate_keyed (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756).
751			*entry = NULL;
752			fsal_release_attrs(&attrs);
753			return status;
754		}
755	
756		status = mdcache_new_entry(export, sub_handle, &attrs, attrs_out,
757					   false, entry, NULL);
758	
759		fsal_release_attrs(&attrs);
760	
(gdb) list *0x684851
0x684851 is in mdcache_create_handle (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700).
1695		key.fsal = sub_export->fsal;
1696	
1697		(void) cih_hash_key(&key, sub_export->fsal, hdl_desc,
1698				    CIH_HASH_KEY_PROTOTYPE);
1699	
1700		status = mdcache_locate_keyed(&key, export, &entry, attrs_out);
1701		if (FSAL_IS_ERROR(status))
1702			return status;
1703	
1704		/* Make sure this entry has a parent pointer */
(gdb) list *0x606683
0x606683 is in nfs3_FhandleToCache (/root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98).
93		    export->exp_ops.extract_handle(export, FSAL_DIGEST_NFSV3,
94						   &fh_desc,
95						   v3_handle->fhflags1);
96	
97		if (!FSAL_IS_ERROR(fsal_status))
98			fsal_status = export->exp_ops.create_handle(export, &fh_desc,
99								    &obj, NULL);
100	
101		if (FSAL_IS_ERROR(fsal_status)) {
102			*status = nfs3_Errno_status(fsal_status);
(gdb) list *0x52e904
0x52e904 is in nfs3_write (/root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124).
119		res->res_write3.WRITE3res_u.resfail.file_wcc.before.attributes_follow =
120		    false;
121		res->res_write3.WRITE3res_u.resfail.file_wcc.after.attributes_follow =
122		    false;
123	
124		obj = nfs3_FhandleToCache(&arg->arg_write3.file,
125					    &res->res_write3.status,
126					    &rc);
127	
128		if (obj == NULL) {
(gdb) 


> freed by thread T208 here:
(gdb) list *0x66a342
0x66a342 is in gsh_free (/root/ravi/nfs-ganesha/src/include/abstract_mem.h:271).
266	 * @param[in] p Block of memory to free.
267	 */
268	static inline void
269	gsh_free(void *p)
270	{
271		free(p);
272	}
273	
274	/**
275	 * @brief Free a block of memory with size
(gdb) list *0x66a3c1
0x66a3c1 is in pool_free (/root/ravi/nfs-ganesha/src/include/abstract_mem.h:420).
415	 */
416	
417	static inline void
418	pool_free(pool_t *pool, void *object)
419	{
420		gsh_free(object);
421	}
422	
423	#endif /* ABSTRACT_MEM_H */
(gdb) list *0x67497e
0x67497e is in mdcache_lru_putback (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1505).
1500			 * are LRU_ENTRY_NONE */
1501			LRU_DQ_SAFE(&entry->lru, q);
1502		}
1503	
1504		/* We do NOT call lru_clean_entry, since it was never initialized. */
1505		pool_free(mdcache_entry_pool, entry);
1506		(void) atomic_dec_int64_t(&lru_state.entries_used);
1507	
1508		if (!qlocked)
1509			QUNLOCK(qlane);
(gdb) list *0x690fd5
0x690fd5 is in mdcache_new_entry (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:582).
577	
578			if (has_hashkey)
579				mdcache_key_delete(&nentry->fh_hk.key);
580	
581			/* Release the new entry we acquired. */
582			mdcache_lru_putback(nentry, LRU_FLAG_NONE);
583		}
584	
585	 out_release:
586	
(gdb) list *0x691caa
0x691caa is in mdcache_locate_keyed (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756).
751			*entry = NULL;
752			fsal_release_attrs(&attrs);
753			return status;
754		}
755	
756		status = mdcache_new_entry(export, sub_handle, &attrs, attrs_out,
757					   false, entry, NULL);
758	
759		fsal_release_attrs(&attrs);
760	
(gdb) list *0x684851
0x684851 is in mdcache_create_handle (/root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700).
1695		key.fsal = sub_export->fsal;
1696	
1697		(void) cih_hash_key(&key, sub_export->fsal, hdl_desc,
1698				    CIH_HASH_KEY_PROTOTYPE);
1699	
1700		status = mdcache_locate_keyed(&key, export, &entry, attrs_out);
1701		if (FSAL_IS_ERROR(status))
1702			return status;
1703	
1704		/* Make sure this entry has a parent pointer */
(gdb) list *0x606683
0x606683 is in nfs3_FhandleToCache (/root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98).
93		    export->exp_ops.extract_handle(export, FSAL_DIGEST_NFSV3,
94						   &fh_desc,
95						   v3_handle->fhflags1);
96	
97		if (!FSAL_IS_ERROR(fsal_status))
98			fsal_status = export->exp_ops.create_handle(export, &fh_desc,
99								    &obj, NULL);
100	
101		if (FSAL_IS_ERROR(fsal_status)) {
102			*status = nfs3_Errno_status(fsal_status);
(gdb) list *0x52e904
0x52e904 is in nfs3_write (/root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124).
119		res->res_write3.WRITE3res_u.resfail.file_wcc.before.attributes_follow =
120		    false;
121		res->res_write3.WRITE3res_u.resfail.file_wcc.after.attributes_follow =
122		    false;
123	
124		obj = nfs3_FhandleToCache(&arg->arg_write3.file,
125					    &res->res_write3.status,
126					    &rc);
127	
128		if (obj == NULL) {
(gdb) list *0x47a255
0x47a255 is in nfs_rpc_execute (/root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281).
1276				&reqdata->r_u.req.svc.rq_xprt->blkin.endp,
1277				"export-id",
1278				(op_ctx->export != NULL)
1279				? op_ctx->export->export_id : -1);
1280	#endif
1281			rc = reqdesc->service_function(arg_nfs, &reqdata->r_u.req.svc,
1282						res_nfs);
1283	
1284	#ifdef USE_LTTNG
1285		tracepoint(nfs_rpc, op_end, reqdata);
(gdb) 


The mdcache entry being used by T52 had been free by T208 thread. There could be two possibilities -

Thread T208 - 

mdcache_new_entry() 

410         /* We did not find the object.  Pull an entry off the LRU. */ 
 411         nentry = mdcache_alloc_handle(export, sub_handle, sub_handle->fs);
 412         if (!nentry) { 
 413                 LogCrit(COMPONENT_CACHE_INODE, "mdcache_alloc_handle failed");      
 414                 status = fsalstat(ERR_FSAL_NOMEM, 0);                 
 415                 goto out_release;                                     
 416         }                                                             
 417         
 418         /* See if someone raced us. */                                
 419         oentry = cih_get_by_key_latch(&key, &latch, CIH_GET_WLOCK, __func__,                                                                     
 420                                         __LINE__);                    
 421         if (oentry) {

...
...
445                 goto out;
 446         }

 562  out:
 563 
 564         if (nentry != NULL) {
 565                 /* We raced or failed, deconstruct the new entry, release
 566                  * the attributes, we may not have copied yet, in which case
 567                  * mask and acl are 0/NULL.
 568                  */
 569                 fsal_release_attrs(&nentry->attrs);
 570 
 571                 /* Destroy the export mapping if any */
 572                 mdc_clean_entry(nentry);
 573 
 574                 /* Destroy the locks */
 575                 PTHREAD_RWLOCK_destroy(&nentry->attr_lock);
 576                 PTHREAD_RWLOCK_destroy(&nentry->content_lock);
 577 
 578                 if (has_hashkey)
 579                         mdcache_key_delete(&nentry->fh_hk.key);
 580

Before this point somehow nentry may have got pulled out of lru by T52 to be used as new entry.
 
 581                 /* Release the new entry we acquired. */
 582                 mdcache_lru_putback(nentry, LRU_FLAG_NONE);
 583         }


Request Dan to confirm the same. Will update the logs/cores generated on other nodes as well.
Comment 8 Soumya Koduri 2016-12-15 16:54:06 UTC 
The stack traces are same in the crashes reported on the other nodes as well.
Comment 9 Daniel Gryniewicz 2016-12-15 19:10:13 UTC 
Created attachment 1232302 [details]
Potential fix?

I've looked at this, and I can't see a real possible race; it should be protected by refcnt.  However, this closes a hole where the qlane lock is dropped temporarily, so it might help?
Comment 10 Ambarish 2016-12-16 07:50:25 UTC 
Dan/Soumya/Jiffin ,

I got Ganesha bits compiled with ASAN with your fix from Jiffin and Soumya,but it does not seem to work.ASAN still complains about "heap-use-after-free" and crashes in the same place.

Copying O/P from stdout :

*******
gqas005
*******

[root@gqas005 ~]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F



=================================================================
==8895== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a00459fc0 at pc 0x41ee67 bp 0x7fe1c594ca30 sp 0x7fe1c594ca20
WRITE of size 8 at 0x604a00459fc0 thread T39
    #0 0x41ee66 in __glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:79
    #1 0x41ef11 in glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:92
    #2 0x41ffc0 in fsal_obj_handle_init /root/ravi/nfs-ganesha/src/FSAL/commonlib.c:183
    #3 0x68dc6e in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:129
    #4 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #5 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #6 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #7 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #8 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #9 0x51d024 in nfs3_create /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_create.c:177
    #10 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #11 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #12 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #13 0x7fe1f2937a97 (/lib64/libasan.so.0+0x19a97)
    #14 0x7fe1f2709dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
    #15 0x7fe1f053273c in __clone (/lib64/libc.so.6+0xf773c)
0x604a00459fc0 is located 64 bytes inside of 1480-byte region [0x604a00459f80,0x604a0045a548)
freed by thread T89 here:
    #0 0x7fe1f2934009 (/lib64/libasan.so.0+0x16009)
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67497d in mdcache_lru_putback /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1491
    #4 0x690fc0 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:582
    #5 0x691c95 in mdcache_locate_keyed /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756
    #6 0x684841 in mdcache_create_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700
    #7 0x606a40 in nfs3_FhandleToCache /root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98
    #8 0x52e904 in nfs3_write /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124
    #9 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #10 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #11 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #12 0x7fe1f2937a97 (/lib64/libasan.so.0+0x19a97)
previously allocated by thread T190 here:
    #0 0x7fe1f2934225 (/lib64/libasan.so.0+0x16225)
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x51d024 in nfs3_create /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_create.c:177
    #12 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #13 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #14 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #15 0x7fe1f2937a97 (/lib64/libasan.so.0+0x19a97)
Thread T39 created by T0 here:
    #0 0x7fe1f2928c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7fe1f045cb34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T89 created by T0 here:
    #0 0x7fe1f2928c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7fe1f045cb34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T190 created by T0 here:
    #0 0x7fe1f2928c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7fe1f045cb34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
SUMMARY: AddressSanitizer: heap-use-after-free /root/ravi/nfs-ganesha/src/include/gsh_list.h:79 __glist_add
Shadow bytes around the buggy address:
  0x0c09c00833a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c00833b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c00833c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c00833d0: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c09c00833e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c00833f0: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c0083400: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0083410: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0083420: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0083430: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0083440: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==8895== ABORTING




***********
On gqas013
***********

[root@gqas013 /]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F



=================================================================
==23178== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a0140c340 at pc 0x41ee67 bp 0x7f1cff51ba30 sp 0x7f1cff51ba20
WRITE of size 8 at 0x604a0140c340 thread T158
    #0 0x41ee66 in __glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:79
    #1 0x41ef11 in glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:92
    #2 0x41ffc0 in fsal_obj_handle_init /root/ravi/nfs-ganesha/src/FSAL/commonlib.c:183
    #3 0x68dc6e in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:129
    #4 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #5 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #6 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #7 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #8 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #9 0x51d024 in nfs3_create /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_create.c:177
    #10 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #11 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #12 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #13 0x7f1d7e5bea97 (/lib64/libasan.so.0+0x19a97)
    #14 0x7f1d7e390dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
    #15 0x7f1d7c1b973c in __clone (/lib64/libc.so.6+0xf773c)
0x604a0140c340 is located 64 bytes inside of 1480-byte region [0x604a0140c300,0x604a0140c8c8)
freed by thread T177 here:
    #0 0x7f1d7e5bb009 (/lib64/libasan.so.0+0x16009)
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67497d in mdcache_lru_putback /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1491
    #4 0x690fc0 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:582
    #5 0x691c95 in mdcache_locate_keyed /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756
    #6 0x684841 in mdcache_create_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700
    #7 0x4ddaa1 in nfs4_mds_putfh /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_putfh.c:211
    #8 0x4de093 in nfs4_op_putfh /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_putfh.c:281
    #9 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #10 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #11 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #12 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #13 0x7f1d7e5bea97 (/lib64/libasan.so.0+0x19a97)
previously allocated by thread T136 here:
    #0 0x7f1d7e5bb225 (/lib64/libasan.so.0+0x16225)
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x4d326c in open4_ex /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1441
    #12 0x4d67fb in nfs4_op_open /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
    #13 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #14 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #15 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #16 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #17 0x7f1d7e5bea97 (/lib64/libasan.so.0+0x19a97)
Thread T158 created by T0 here:
    #0 0x7f1d7e5afc3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f1d7c0e3b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T177 created by T0 here:
    #0 0x7f1d7e5afc3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f1d7c0e3b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T136 created by T0 here:
    #0 0x7f1d7e5afc3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f1d7c0e3b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
SUMMARY: AddressSanitizer: heap-use-after-free /root/ravi/nfs-ganesha/src/include/gsh_list.h:79 __glist_add
Shadow bytes around the buggy address:
  0x0c09c0279810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0279820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0279830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0279840: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c09c0279850: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c0279860: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c0279870: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0279880: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0279890: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c02798a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c02798b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==23178== ABORTING



***********
On gqas006
***********

[root@gqas006 ~]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F



=================================================================
==23717== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a0098cdc0 at pc 0x41ee67 bp 0x7f5c4e4861c0 sp 0x7f5c4e4861b0
WRITE of size 8 at 0x604a0098cdc0 thread T215
    #0 0x41ee66 in __glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:79
    #1 0x41ef11 in glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:92
    #2 0x41ffc0 in fsal_obj_handle_init /root/ravi/nfs-ganesha/src/FSAL/commonlib.c:183
    #3 0x68dc6e in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:129
    #4 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #5 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #6 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #7 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #8 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #9 0x4d326c in open4_ex /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1441
    #10 0x4d67fb in nfs4_op_open /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
    #11 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #12 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #13 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #14 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #15 0x7f5cf49f1a97 (/lib64/libasan.so.0+0x19a97)
    #16 0x7f5cf47c3dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
    #17 0x7f5cf25ec73c in __clone (/lib64/libc.so.6+0xf773c)
0x604a0098cdc0 is located 64 bytes inside of 1480-byte region [0x604a0098cd80,0x604a0098d348)
freed by thread T178 here:
    #0 0x7f5cf49ee009 (/lib64/libasan.so.0+0x16009)
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67497d in mdcache_lru_putback /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1491
    #4 0x690fc0 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:582
    #5 0x691c95 in mdcache_locate_keyed /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756
    #6 0x684841 in mdcache_create_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700
    #7 0x606a40 in nfs3_FhandleToCache /root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98
    #8 0x52e904 in nfs3_write /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124
    #9 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #10 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #11 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #12 0x7f5cf49f1a97 (/lib64/libasan.so.0+0x19a97)
previously allocated by thread T211 here:
    #0 0x7f5cf49ee225 (/lib64/libasan.so.0+0x16225)
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x4d326c in open4_ex /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1441
    #12 0x4d67fb in nfs4_op_open /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
    #13 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #14 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #15 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #16 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #17 0x7f5cf49f1a97 (/lib64/libasan.so.0+0x19a97)
Thread T215 created by T0 here:
    #0 0x7f5cf49e2c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f5cf2516b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T178 created by T0 here:
    #0 0x7f5cf49e2c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f5cf2516b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T211 created by T0 here:
    #0 0x7f5cf49e2c3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f5cf2516b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
SUMMARY: AddressSanitizer: heap-use-after-free /root/ravi/nfs-ganesha/src/include/gsh_list.h:79 __glist_add
Shadow bytes around the buggy address:
  0x0c09c0129960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0129970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0129980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c09c0129990: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c09c01299a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c01299b0: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c01299c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c01299d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c01299e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c01299f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c0129a00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe




***********
On gqas011
***********

[root@gqas011 ~]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F



=================================================================
==25530== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a01dda040 at pc 0x41ee67 bp 0x7f0e19315e30 sp 0x7f0e19315e20
WRITE of size 8 at 0x604a01dda040 thread T28
    #0 0x41ee66 in __glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:79
    #1 0x41ef11 in glist_add /root/ravi/nfs-ganesha/src/include/gsh_list.h:92
    #2 0x41ffc0 in fsal_obj_handle_init /root/ravi/nfs-ganesha/src/FSAL/commonlib.c:183
    #3 0x68dc6e in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:129
    #4 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #5 0x691c95 in mdcache_locate_keyed /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756
    #6 0x684841 in mdcache_create_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700
    #7 0x606a40 in nfs3_FhandleToCache /root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98
    #8 0x52e904 in nfs3_write /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124
    #9 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #10 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #11 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #12 0x7f0e3e9a8a97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
    #13 0x7f0e3e77adc4 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308
    #14 0x7f0e3c5a373c in __clone (/lib64/libc.so.6+0xf773c)
0x604a01dda040 is located 64 bytes inside of 1480-byte region [0x604a01dda000,0x604a01dda5c8)
freed by thread T231 here:
    #0 0x7f0e3e9a5009 in __interceptor_free /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_malloc_linux.cc:61
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67497d in mdcache_lru_putback /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1491
    #4 0x690fc0 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:582
    #5 0x691c95 in mdcache_locate_keyed /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:756
    #6 0x684841 in mdcache_create_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1700
    #7 0x606a40 in nfs3_FhandleToCache /root/ravi/nfs-ganesha/src/support/nfs_filehandle_mgmt.c:98
    #8 0x52e904 in nfs3_write /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_write.c:124
    #9 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #10 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #11 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #12 0x7f0e3e9a8a97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
previously allocated by thread T67 here:
    #0 0x7f0e3e9a5225 in __interceptor_calloc /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_malloc_linux.cc:87
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x4d326c in open4_ex /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1441
    #12 0x4d67fb in nfs4_op_open /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
    #13 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #14 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #15 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #16 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #17 0x7f0e3e9a8a97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
Thread T28 created by T0 here:
    #0 0x7f0e3e999c3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f0e3c4cdb34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
Thread T231 created by T0 here:
    #0 0x7f0e3e999c3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f0e3c4cdb34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
Thread T67 created by T0 here:
    #0 0x7f0e3e999c3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f0e3c4cdb34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
SUMMARY: AddressSanitizer: heap-use-after-free /root/ravi/nfs-ganesha/src/include/gsh_list.h:79 __glist_add
Shadow bytes around the buggy address:
  0x0c09c03b33b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c03b33c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c03b33d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c03b33e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c09c03b33f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c09c03b3400: fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd
  0x0c09c03b3410: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c03b3420: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c03b3430: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c03b3440: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c09c03b3450: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==25530== ABORTING
Comment 11 Soumya Koduri 2016-12-16 10:18:42 UTC 
While we wait for Dan's input, I have provided following patch to test once -

diff --git a/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c b/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_
index 66cb31d..c12edc1 100644
--- a/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c
+++ b/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c
@@ -562,6 +562,7 @@ mdcache_new_entry(struct mdcache_fsal_export *export,
  out:
 
        if (nentry != NULL) {
+#ifdef crash_fix
                /* We raced or failed, deconstruct the new entry, release
                 * the attributes, we may not have copied yet, in which case
                 * mask and acl are 0/NULL.
@@ -580,6 +581,8 @@ mdcache_new_entry(struct mdcache_fsal_export *export,
 
                /* Release the new entry we acquired. */
                mdcache_lru_putback(nentry);
+#endif
+               (void)mdcache_kill_entry(nentry);
        }
 
taking similar approach of purging a cache entry when a file is deleted/becomes stale.
Comment 12 Ambarish 2016-12-17 04:18:38 UTC 
Hitting a crash in glist_del :

ASAN Report :

************
On gqas006
************
root@gqas006:/[root@gqas006 /]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F

==6314== WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
=================================================================
==6314== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a0114daa0 at pc 0x5a1a95 bp 0x7ffa8767b730 sp 0x7ffa8767b720
WRITE of size 8 at 0x604a0114daa0 thread T178
    #0 0x5a1a94 in glist_del /root/ravi/nfs-ganesha/src/include/gsh_list.h:101
    #1 0x5a5626 in state_del_locked /root/ravi/nfs-ganesha/src/SAL/nfs4_state.c:373
    #2 0x4a652e in nfs4_op_close /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_close.c:310
    #3 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #4 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #5 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #6 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #7 0x7ffb1249ea97 (/lib64/libasan.so.0+0x19a97)
    #8 0x7ffb12270dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
    #9 0x7ffb1009973c in __clone (/lib64/libc.so.6+0xf773c)
0x604a0114daa0 is located 1184 bytes inside of 1480-byte region [0x604a0114d600,0x604a0114dbc8)
freed by thread T233 here:
    #0 0x7ffb1249b009 (/lib64/libasan.so.0+0x16009)
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67458b in mdcache_lru_unref /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1456
    #4 0x679e2d in mdcache_put /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h:186
    #5 0x683134 in mdcache_put_ref /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1472
    #6 0x44dda1 in fsal_remove /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1599
    #7 0x4eb2d0 in nfs4_op_remove /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_remove.c:104
    #8 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #9 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #10 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #11 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #12 0x7ffb1249ea97 (/lib64/libasan.so.0+0x19a97)
previously allocated by thread T180 here:
    #0 0x7ffb1249b225 (/lib64/libasan.so.0+0x16225)
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x4d326c in open4_ex /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1441
    #12 0x4d67fb in nfs4_op_open /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_open.c:1844
    #13 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #14 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #15 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #16 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #17 0x7ffb1249ea97 (/lib64/libasan.so.0+0x19a97)
Thread T178 created by T0 here:
    #0 0x7ffb1248fc3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7ffb0ffc3b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T233 created by T0 here:
    #0 0x7ffb1248fc3a (/lib64/libasan.so.0+0xac3a)
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7ffb0ffc3b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T180 created by T0 here:
    #0 0x7ffb1248fc3a (/lib64/libasan.so.0+0xac3a)
[root@gqas006 /]# exit


**********
On gqas011
**********

kroot@gqas011:/[root@gqas011 /]# /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F
==21912== WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
cat ty	
=================================================================
==21912== ERROR: AddressSanitizer: heap-use-after-free on address 0x604a009627a0 at pc 0x5a1a95 bp 0x7f62e3b2a730 sp 0x7f62e3b2a720
WRITE of size 8 at 0x604a009627a0 thread T132
    #0 0x5a1a94 in glist_del /root/ravi/nfs-ganesha/src/include/gsh_list.h:101
    #1 0x5a5626 in state_del_locked /root/ravi/nfs-ganesha/src/SAL/nfs4_state.c:373
    #2 0x4a652e in nfs4_op_close /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_op_close.c:310
    #3 0x4a2147 in nfs4_Compound /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:734
    #4 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #5 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #6 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #7 0x7f634eddda97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
    #8 0x7f634ebafdc4 in start_thread /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:308
    #9 0x7f634c9d873c in __clone (/lib64/libc.so.6+0xf773c)
0x604a009627a0 is located 1184 bytes inside of 1480-byte region [0x604a00962300,0x604a009628c8)
freed by thread T113 here:
    #0 0x7f634edda009 in __interceptor_free /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_malloc_linux.cc:61
    #1 0x66a6ff in gsh_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:271
    #2 0x66a77e in pool_free /root/ravi/nfs-ganesha/src/include/abstract_mem.h:420
    #3 0x67458b in mdcache_lru_unref /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1456
    #4 0x679e2d in mdcache_put /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h:186
    #5 0x683134 in mdcache_put_ref /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1472
    #6 0x52a832 in nfs3_remove /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_remove.c:162
    #7 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #8 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #9 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #10 0x7f634eddda97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
previously allocated by thread T215 here:
    #0 0x7f634edda225 in __interceptor_calloc /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_malloc_linux.cc:87
    #1 0x66a6b4 in gsh_calloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x66a760 in pool_alloc__ /root/ravi/nfs-ganesha/src/include/abstract_mem.h:395
    #3 0x6720aa in alloc_cache_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1157
    #4 0x672358 in mdcache_lru_get /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1197
    #5 0x68da9a in mdcache_alloc_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:117
    #6 0x690013 in mdcache_new_entry /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:411
    #7 0x67a23a in mdcache_alloc_and_check_handle /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:94
    #8 0x687d1d in mdcache_open2 /root/ravi/nfs-ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c:702
    #9 0x446cc5 in open2_by_name /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:399
    #10 0x44f49f in fsal_open2 /root/ravi/nfs-ganesha/src/FSAL/fsal_helper.c:1816
    #11 0x51d024 in nfs3_create /root/ravi/nfs-ganesha/src/Protocols/NFS/nfs3_create.c:177
    #12 0x47a255 in nfs_rpc_execute /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1281
    #13 0x47b9d3 in worker_run /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1548
    #14 0x6257db in fridgethr_start_routine /root/ravi/nfs-ganesha/src/support/fridgethr.c:550
    #15 0x7f634eddda97 in __asan::AsanThread::ThreadStart() /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_thread.cc:99
Thread T132 created by T0 here:
    #0 0x7f634edcec3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f634c902b34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
Thread T113 created by T0 here:
    #0 0x7f634edcec3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f634c902b34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
Thread T215 created by T0 here:
    #0 0x7f634edcec3a in __interceptor_pthread_create /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libsanitizer/asan/../../../../libsanitizer/asan/asan_interceptors.cc:122
    #1 0x62d8e2 in fridgethr_populate /root/ravi/nfs-ganesha/src/support/fridgethr.c:1418
    #2 0x47bf46 in worker_init /root/ravi/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1615
    #3 0x48d94e in nfs_Start_threads /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:457
    #4 0x48fbc4 in nfs_start /root/ravi/nfs-ganesha/src/MainNFSD/nfs_init.c:879
    #5 0x41d81d in main /root/ravi/nfs-ganesha/src/MainNFSD/nfs_main.c:470
    #6 0x7f634c902b34 in __libc_start_main /usr/src/debug/glibc-2.17-c758a686/csu/../csu/libc-start.c:274
SUMMARY: AddressSanitizer: heap-use-after-free /root/ravi/nfs-ganesha/src/include/gsh_list.h:101 glist_del
Comment 13 Ravishankar N 2016-12-20 07:03:50 UTC 
Reassigning the BZ to ganesha folks for now. Feel free to assign it back to replicate if we hit an issue in AFR.
Comment 17 Ambarish 2017-01-20 07:53:29 UTC 
The reported issue was not reproducible on Ganesha 2.4.1-6,Gluster 3.8.4-12 on two tries.

Will reopen if hit again during regressions.
Comment 19 errata-xmlrpc 2017-03-23 06:26:29 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2017-0493.html
Note You need to log in before you can comment on or make changes to this bug.