Description of problem: While stopping and deleting volume on which ganesha was enabled glusterd crashed Version-Release number of selected component (if applicable): glusterfs-ganesha-3.8.4-24.el7rhgs.x86_64 How reproducible: Steps to Reproduce: 1.Create a 7 node ganesha cluster 2.Create 3*2 Distributed-Replicate Volumes.Enable ganesha on it 3.Stop and delete Volume 1 Actual results: Volume should be able to delete Expected results: Deleting volume failed with staging failed message and glusterd crashed on 5 out of 7 nodes Additional info: I was Unable to capture the core dumps for 1st volume as the cores were getting deleted automatically from /var/spool/abrt folder. I did settings to enable core on my setup. Attaching the core generated while deleting volume2. pending frames: frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2017-04-28 13:09:16 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.8.4 /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f2e433170e2] /lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f2e43320b04] /lib64/libc.so.6(+0x35250)[0x7f2e419f9250] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x9864b)[0x7f2e37eb364b] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xc2ee3)[0x7f2e37eddee3] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x37cac)[0x7f2e37e52cac] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x4149c)[0x7f2e37e5c49c] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3e40f)[0x7f2e37e5940f] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x1a9c2)[0x7f2e37e359c2] /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x20dee)[0x7f2e37e3bdee] /lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f2e4334d082] /lib64/libc.so.6(+0x46cf0)[0x7f2e41a0acf0]
t a a bt output please?
Continuing. Detaching after fork from child process 15545. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fb589d0f700 (LWP 10714)] 0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so (gdb) bt #0 0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #1 0x00007fb589256ee3 in glusterd_op_delete_volume () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #2 0x00007fb5891cbcac in glusterd_op_commit_perform () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #3 0x00007fb5891d549c in glusterd_op_ac_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #4 0x00007fb5891d240f in glusterd_op_sm () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #5 0x00007fb5891ae9c2 in __glusterd_handle_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #6 0x00007fb5891b4dee in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #7 0x00007fb5946c6082 in synctask_wrap () from /lib64/libglusterfs.so.0 #8 0x00007fb592d83cf0 in ?? () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () (gdb) t a a bt Thread 8 (Thread 0x7fb58bd13700 (LWP 10710)): #0 0x00007fb5934f6bdd in nanosleep () from /lib64/libpthread.so.0 #1 0x00007fb59469d24e in gf_timer_proc () from /lib64/libglusterfs.so.0 #2 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fb592e3473d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fb58b512700 (LWP 10711)): #0 0x00007fb5934f7101 in sigwait () from /lib64/libpthread.so.0 #1 0x00007fb594b85eab in glusterfs_sigwaiter () #2 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fb592e3473d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fb58ad11700 (LWP 10712)): #0 0x00007fb592dfb66d in nanosleep () from /lib64/libc.so.6 #1 0x00007fb592dfb504 in sleep () from /lib64/libc.so.6 #2 0x00007fb5946b659d in pool_sweeper () from /lib64/libglusterfs.so.0 #3 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb592e3473d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fb58a510700 (LWP 10713)): #0 0x00007fb5934f3a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb5946c85b8 in syncenv_task () from /lib64/libglusterfs.so.0 #2 0x00007fb5946c9400 in syncenv_processor () from /lib64/libglusterfs.so.0 #3 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fb592e3473d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 4 (Thread 0x7fb589d0f700 (LWP 10714)): #0 0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #1 0x00007fb589256ee3 in glusterd_op_delete_volume () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #2 0x00007fb5891cbcac in glusterd_op_commit_perform () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #3 0x00007fb5891d549c in glusterd_op_ac_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #4 0x00007fb5891d240f in glusterd_op_sm () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #5 0x00007fb5891ae9c2 in __glusterd_handle_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #6 0x00007fb5891b4dee in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #7 0x00007fb5946c6082 in synctask_wrap () from /lib64/libglusterfs.so.0 #8 0x00007fb592d83cf0 in ?? () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? () Thread 3 (Thread 0x7fb584e7a700 (LWP 10989)): #0 0x00007fb5934f36d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fb589268afb in hooks_worker () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so #2 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fb592e3473d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fb57ffff700 (LWP 10990)): #0 0x00007fb592e34d13 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fb5946e9ce0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0 #2 0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fb592e3473d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fb594b68780 (LWP 10709)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007fb5934f0ef7 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007fb5946ea290 in event_dispatch_epoll () from /lib64/libglusterfs.so.0 #2 0x00007fb594b82d93 in main ()
Downstream patch link https://code.engineering.redhat.com/gerrit/#/c/104975/
Verified this bug on glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64 glusterd no more crash while doing volume stop and delete on which ganesha is enable.Hence moving this bug to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774