Description of problem: ------------------------ I have a 100 EC volumes exported via Ganesha. 2 of them are active - butcher1 and butcher2. There is Bonnie and dbench running via v3 and v4 on these two exports. The other 98 exports are passive. I was exporting/unepxorting these 98 passive volumes at random (via vol restarts and ganesha.enable on/off). Ganesha crashed on one the nodes and dumped a core in the meantime . This is what I could fetch from the core : Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.c'. Program terminated with signal 11, Segmentation fault. #0 0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315 315 GF_ASSERT (GF_MEM_TRAILER_MAGIC == Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-7.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 elfutils-libs-0.170-4.el7.x86_64 glibc-2.17-222.el7.x86_64 gssproxy-0.7.0-17.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-11.el7.x86_64 libgcc-4.8.5-28.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 sssd-client-1.16.0-19.el7.x86_64 systemd-libs-219-57.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f1716e0360d in __gf_free (free_ptr=0x7f16a4adf5d0) at mem-pool.c:315 #1 0x00007f1716de912a in gf_timer_proc (data=0x7f171033eb00) at timer.c:157 #2 0x00007f17a8151dd5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f17a781db3d in clone () from /lib64/libc.so.6 (gdb) Version-Release number of selected component (if applicable): -------------------------------------------------------------- glusterfs-ganesha-3.12.2-6.el7rhgs.x86_64 nfs-ganesha-2.5.5-3.el7rhgs.x86_64 How reproducible: ----------------- 1/1 Additional info:
REVISION POSTED: https://review.gluster.org/19813 (libglusterfs/timer: race condition and mem access after free) posted (#2) for review on master by Kaleb KEITHLEY