Bug 1049166

Description of problem:
=======================

Have observed a glusterd crash while taking volume status from one node in cluster when the other nodes were performing stop and start of glusterd.

My setup consist of

1.Four servers: server1,server2,server3 and server4 in a cluster.
2.Four volumes: vol0,vol1,vol2,vol3

Stopped the glusterd on server3 and server4. Checked the gluster volume status which was successful. Again started the glusterd on server3 and server4, while glusterd was starting, tried to take the volume status from server1 and immediately glusterd on server3 crashed with bt as follows:

(gdb) bt
#0  0x0000003f5247873b in _int_free () from /lib64/libc.so.6
#1  0x0000003543a183b8 in dict_destroy (this=0x7fdf6b5a183c) at dict.c:454
#2  0x00007fdf692fe818 in glusterd_op_fini_ctx () at glusterd-op-sm.c:6180
#3  0x00007fdf69306801 in glusterd_op_ac_commit_op (event=0x7fdf58000cb0, ctx=0x7fdf580009e0) at glusterd-op-sm.c:4247
#4  0x00007fdf693030d0 in glusterd_op_sm () at glusterd-op-sm.c:6047
#5  0x00007fdf692ed45b in __glusterd_handle_commit_op (req=0x7fdf6926102c) at glusterd-handler.c:1006
#6  0x00007fdf692ea3cf in glusterd_big_locked_handler (req=0x7fdf6926102c, actor_fn=0x7fdf692ed350 <__glusterd_handle_commit_op>) at glusterd-handler.c:78
#7  0x0000003543a4cdd2 in synctask_wrap (old_task=<value optimized out>) at syncop.c:293
#8  0x0000003f52443bf0 in ?? () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()
 


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.4.0.snap.dec30.2013git-1.el6.x86_64

Steps carried:
==============
1. Create a setup of four servers (server1-4)
2. Create four volumes (vol0,vol1,vol2 and vol3)
3. Stop glusterd on server3 and server4
4. Check the volume status on server1
5. Start the glusterd on server3 and server4
6. While start in progress, take the volume status from server1


Actual results:
===============

glusterd crashed with bt mentioned above, and logs mentioned below:

pending frames:
frame : type(0) op(0)

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-01-07 00:01:36configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.4.0.snap.dec30.2013git
/lib64/libc.so.6[0x3f524329a0]
/lib64/libc.so.6[0x3f5247873b]
/usr/lib64/libglusterfs.so.0(dict_destroy+0x48)[0x3543a183b8]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_fini_ctx+0x18)[0x7fdf692fe818]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(+0x45801)[0x7fdf69306801]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_sm+0x1f0)[0x7fdf693030d0]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(__glusterd_handle_commit_op+0x10b)[0x7fdf692ed45b]
/usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fdf692ea3cf]
/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x3543a4cdd2]
/lib64/libc.so.6[0x3f52443bf0]
---------


Expected results:
=================

glusterd should not crash