Bug 1446645

Summary: [GANESHA] Glusterd crashed while deleting volume on which ganesha was enabled
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Manisha Saini <msaini>
Component: glusterdAssignee: Jiffin <jthottan>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: rhgs-3.3CC: amukherj, msaini, rhinduja, rhs-bugs, skoduri, storage-qa-internal, vbellur
Target Milestone: ---   
Target Release: RHGS 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8.4-25 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-21 04:41:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1417151    

Description Manisha Saini 2017-04-28 13:48:31 UTC 
Description of problem:

While stopping and deleting volume on which ganesha was enabled glusterd crashed 

Version-Release number of selected component (if applicable):

glusterfs-ganesha-3.8.4-24.el7rhgs.x86_64

How reproducible:


Steps to Reproduce:
1.Create a 7 node ganesha cluster
2.Create 3*2 Distributed-Replicate Volumes.Enable ganesha on it
3.Stop and delete Volume 1


Actual results:
Volume should be able to delete

Expected results:
Deleting volume failed with staging failed message and glusterd crashed on 5 out of 7 nodes

Additional info:

I was Unable to capture the core dumps for 1st volume as the cores were getting deleted automatically from /var/spool/abrt folder.

I did settings to enable core on my setup.
Attaching the core generated while deleting volume2.


pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2017-04-28 13:09:16
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8.4
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xc2)[0x7f2e433170e2]
/lib64/libglusterfs.so.0(gf_print_trace+0x324)[0x7f2e43320b04]
/lib64/libc.so.6(+0x35250)[0x7f2e419f9250]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x9864b)[0x7f2e37eb364b]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xc2ee3)[0x7f2e37eddee3]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x37cac)[0x7f2e37e52cac]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x4149c)[0x7f2e37e5c49c]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x3e40f)[0x7f2e37e5940f]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x1a9c2)[0x7f2e37e359c2]
/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x20dee)[0x7f2e37e3bdee]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f2e4334d082]
/lib64/libc.so.6(+0x46cf0)[0x7f2e41a0acf0]
Comment 3 Atin Mukherjee 2017-04-28 15:32:28 UTC 
t a a bt output please?
Comment 4 Manisha Saini 2017-04-28 17:00:53 UTC 
Continuing.
Detaching after fork from child process 15545.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb589d0f700 (LWP 10714)]
0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
(gdb) bt
#0  0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#1  0x00007fb589256ee3 in glusterd_op_delete_volume () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#2  0x00007fb5891cbcac in glusterd_op_commit_perform () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#3  0x00007fb5891d549c in glusterd_op_ac_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#4  0x00007fb5891d240f in glusterd_op_sm () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#5  0x00007fb5891ae9c2 in __glusterd_handle_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#6  0x00007fb5891b4dee in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#7  0x00007fb5946c6082 in synctask_wrap () from /lib64/libglusterfs.so.0
#8  0x00007fb592d83cf0 in ?? () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()
(gdb) t a a bt

Thread 8 (Thread 0x7fb58bd13700 (LWP 10710)):
#0  0x00007fb5934f6bdd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007fb59469d24e in gf_timer_proc () from /lib64/libglusterfs.so.0
#2  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fb592e3473d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fb58b512700 (LWP 10711)):
#0  0x00007fb5934f7101 in sigwait () from /lib64/libpthread.so.0
#1  0x00007fb594b85eab in glusterfs_sigwaiter ()
#2  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fb592e3473d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fb58ad11700 (LWP 10712)):
#0  0x00007fb592dfb66d in nanosleep () from /lib64/libc.so.6
#1  0x00007fb592dfb504 in sleep () from /lib64/libc.so.6
#2  0x00007fb5946b659d in pool_sweeper () from /lib64/libglusterfs.so.0
#3  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fb592e3473d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fb58a510700 (LWP 10713)):
#0  0x00007fb5934f3a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fb5946c85b8 in syncenv_task () from /lib64/libglusterfs.so.0
#2  0x00007fb5946c9400 in syncenv_processor () from /lib64/libglusterfs.so.0
#3  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fb592e3473d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 4 (Thread 0x7fb589d0f700 (LWP 10714)):
#0  0x00007fb58922c64b in manage_export_config () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#1  0x00007fb589256ee3 in glusterd_op_delete_volume () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#2  0x00007fb5891cbcac in glusterd_op_commit_perform () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#3  0x00007fb5891d549c in glusterd_op_ac_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#4  0x00007fb5891d240f in glusterd_op_sm () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#5  0x00007fb5891ae9c2 in __glusterd_handle_commit_op () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#6  0x00007fb5891b4dee in glusterd_big_locked_handler () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#7  0x00007fb5946c6082 in synctask_wrap () from /lib64/libglusterfs.so.0
#8  0x00007fb592d83cf0 in ?? () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fb584e7a700 (LWP 10989)):
#0  0x00007fb5934f36d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fb589268afb in hooks_worker () from /usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so
#2  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fb592e3473d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fb57ffff700 (LWP 10990)):
#0  0x00007fb592e34d13 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fb5946e9ce0 in event_dispatch_epoll_worker () from /lib64/libglusterfs.so.0
#2  0x00007fb5934efdc5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fb592e3473d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fb594b68780 (LWP 10709)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007fb5934f0ef7 in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fb5946ea290 in event_dispatch_epoll () from /lib64/libglusterfs.so.0
#2  0x00007fb594b82d93 in main ()
Comment 6 Jiffin 2017-05-03 06:12:17 UTC 
Downstream patch link https://code.engineering.redhat.com/gerrit/#/c/104975/
Comment 10 Manisha Saini 2017-05-26 12:16:31 UTC 
Verified this bug on glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64


glusterd no more crash while doing volume stop and delete on which ganesha is enable.Hence moving this bug to verified state.
Comment 12 errata-xmlrpc 2017-09-21 04:41:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774