Description of problem:
=======================

Have observed a glusterd crash while taking volume status from one node in cluster when the other nodes were performing stop and start of glusterd.

My setup consist of

1.Four servers: server1,server2,server3 and server4 in a cluster.
2.Four volumes: vol0,vol1,vol2,vol3

Stopped the glusterd on server3 and server4. Checked the gluster volume status which was successful. Again started the glusterd on server3 and server4, while glusterd was starting, tried to take the volume status from server1 and immediately glusterd on server2 crashed with bt as follows:

(gdb) bt
#0  0x000000321340c380 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x0000003f09c3f43f in mem_put (ptr=0x7fd2e1d17814) at mem-pool.c:484
#2  0x0000003f09c183c5 in dict_destroy (this=0x7fd2e1d17c34) at dict.c:456
#3  0x00007fd2dfa75818 in glusterd_op_fini_ctx () at glusterd-op-sm.c:6180
#4  0x00007fd2dfa7d801 in glusterd_op_ac_commit_op (event=0x7fd2d4016690, ctx=0x7fd2d4016490) at glusterd-op-sm.c:4247
#5  0x00007fd2dfa7a0d0 in glusterd_op_sm () at glusterd-op-sm.c:6047
#6  0x00007fd2dfa6445b in __glusterd_handle_commit_op (req=0x7fd2df9d802c) at glusterd-handler.c:1006
#7  0x00007fd2dfa613cf in glusterd_big_locked_handler (req=0x7fd2df9d802c, actor_fn=0x7fd2dfa64350 <__glusterd_handle_commit_op>) at glusterd-handler.c:78
#8  0x0000003f09c4cdd2 in synctask_wrap (old_task=<value optimized out>) at syncop.c:293
#9  0x0000003213043bf0 in ?? () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()
(gdb) 

 


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.4.0.snap.dec30.2013git-1.el6.x86_64

Steps carried:
==============
1. Create a setup of four servers (server1-4)
2. Create four volumes (vol0,vol1,vol2 and vol3)
3. Stop glusterd on server3 and server4
4. Check the volume status on server1
5. Start the glusterd on server3 and server4
6. While start in progress, take the volume status from server1


Actual results:
===============

glusterd crashed with bt mentioned above, and logs mentioned below:

pending frames:
frame : type(0) op(0)
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-01-07 00:01:34configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.snap.dec30.2013git
/lib64/libc.so.6[0x32130329a0]
/lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x321340c380]
/usr/lib64/libglusterfs.so.0(mem_put+0x6f)[0x3f09c3f43f]
/usr/lib64/libglusterfs.so.0(dict_destroy+0x55)[0x3f09c183c5]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_fini_ctx+0x18)[0x7fd2dfa75818]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(+0x45801)[0x7fd2dfa7d801]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_sm+0x1f0)[0x7fd2dfa7a0d0]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(__glusterd_handle_commit_op+0x10b)[0x7fd2dfa6445b]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fd2dfa613cf]
/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x3f09c4cdd2]
/lib64/libc.so.6[0x3213043bf0]
---------


Expected results:
=================

glusterd should not crash