Bug 1562769 - [Ganesha] : Ganesha crashed in gf_timer_proc().
Summary: [Ganesha] : Ganesha crashed in gf_timer_proc().
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: nfs-ganesha
Version: rhgs-3.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Kaleb KEITHLEY
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On: 1562951 1563306
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-02 13:02 UTC by Ambarish
Modified: 2018-11-19 10:32 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-19 10:32:05 UTC
Embargoed:


Attachments (Terms of Use)

Description Ambarish 2018-04-02 13:02:43 UTC
Description of problem:
------------------------

I have a 100 EC volumes exported via Ganesha.

2 of them are active - butcher1 and butcher2.

There is Bonnie and dbench running via v3 and v4 on these two exports.

The other 98 exports are passive.

I was exporting/unepxorting these 98 passive volumes at random (via vol restarts and ganesha.enable on/off).


Ganesha crashed on one the nodes and dumped a core in the meantime .

This is what I could fetch from the core :


Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315
315	        GF_ASSERT (GF_MEM_TRAILER_MAGIC ==
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-7.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64 gssproxy-0.7.0-17.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-11.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 sssd-client-1.16.0-19.el7.x86_64 systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315
#1  0x00007f1716de912a in gf_timer_proc (data=0x7f171033eb00) at timer.c:157
#2  0x00007f17a8151dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f17a781db3d in clone () from /lib64/libc.so.6
(gdb) 




Version-Release number of selected component (if applicable):
--------------------------------------------------------------

glusterfs-ganesha-3.12.2-6.el7rhgs.x86_64
nfs-ganesha-2.5.5-3.el7rhgs.x86_64


How reproducible:
-----------------

1/1



Additional info:

Comment 3 Worker Ant 2018-04-03 14:49:21 UTC
REVISION POSTED: https://review.gluster.org/19813 (libglusterfs/timer: race condition and mem access after free) posted (#2) for review on master by Kaleb KEITHLEY


Note You need to log in before you can comment on or make changes to this bug.