Bug 1562769

Summary: [Ganesha] : Ganesha crashed in gf_timer_proc().
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ambarish <asoman>
Component: nfs-ganeshaAssignee: Kaleb KEITHLEY <kkeithle>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Manisha Saini <msaini>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: bturner, dang, ffilz, jthottan, kkeithle, mbenjamin, pasik, rhinduja, rhs-bugs, sheggodu, skoduri, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-19 10:32:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1562951, 1563306    
Bug Blocks:    

Description Ambarish 2018-04-02 13:02:43 UTC 
Description of problem:
------------------------

I have a 100 EC volumes exported via Ganesha.

2 of them are active - butcher1 and butcher2.

There is Bonnie and dbench running via v3 and v4 on these two exports.

The other 98 exports are passive.

I was exporting/unepxorting these 98 passive volumes at random (via vol restarts and ganesha.enable on/off).


Ganesha crashed on one the nodes and dumped a core in the meantime .

This is what I could fetch from the core :


Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315
315	        GF_ASSERT (GF_MEM_TRAILER_MAGIC ==
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-7.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64 gssproxy-0.7.0-17.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-11.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 sssd-client-1.16.0-19.el7.x86_64 systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315
#1  0x00007f1716de912a in gf_timer_proc (data=0x7f171033eb00) at timer.c:157
#2  0x00007f17a8151dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f17a781db3d in clone () from /lib64/libc.so.6
(gdb) 




Version-Release number of selected component (if applicable):
--------------------------------------------------------------

glusterfs-ganesha-3.12.2-6.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64


How reproducible:
-----------------

1/1



Additional info:
Comment 3 Worker Ant 2018-04-03 14:49:21 UTC 
REVISION POSTED: https://review.gluster.org/19813 (libglusterfs/timer: race condition and mem access after free) posted (#2) for review on master by Kaleb KEITHLEY