Description of problem: ======================= Have observed a glusterd crash while taking volume status from one node in cluster when the other nodes were performing stop and start of glusterd. My setup consist of 1.Four servers: server1,server2,server3 and server4 in a cluster. 2.Four volumes: vol0,vol1,vol2,vol3 Stopped the glusterd on server3 and server4. Checked the gluster volume status which was successful. Again started the glusterd on server3 and server4, while glusterd was starting, tried to take the volume status from server1 and immediately glusterd on server2 crashed with bt as follows: (gdb) bt #0 0x000000321340c380 in pthread_spin_lock () from /lib64/libpthread.so.0 #1 0x0000003f09c3f43f in mem_put (ptr=0x7fd2e1d17814) at mem-pool.c:484 #2 0x0000003f09c183c5 in dict_destroy (this=0x7fd2e1d17c34) at dict.c:456 #3 0x00007fd2dfa75818 in glusterd_op_fini_ctx () at glusterd-op-sm.c:6180 #4 0x00007fd2dfa7d801 in glusterd_op_ac_commit_op (event=0x7fd2d4016690, ctx=0x7fd2d4016490) at glusterd-op-sm.c:4247 #5 0x00007fd2dfa7a0d0 in glusterd_op_sm () at glusterd-op-sm.c:6047 #6 0x00007fd2dfa6445b in __glusterd_handle_commit_op (req=0x7fd2df9d802c) at glusterd-handler.c:1006 #7 0x00007fd2dfa613cf in glusterd_big_locked_handler (req=0x7fd2df9d802c, actor_fn=0x7fd2dfa64350 <__glusterd_handle_commit_op>) at glusterd-handler.c:78 #8 0x0000003f09c4cdd2 in synctask_wrap (old_task=<value optimized out>) at syncop.c:293 #9 0x0000003213043bf0 in ?? () from /lib64/libc.so.6 #10 0x0000000000000000 in ?? () (gdb) Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.4.0.snap.dec30.2013git-1.el6.x86_64 Steps carried: ============== 1. Create a setup of four servers (server1-4) 2. Create four volumes (vol0,vol1,vol2 and vol3) 3. Stop glusterd on server3 and server4 4. Check the volume status on server1 5. Start the glusterd on server3 and server4 6. While start in progress, take the volume status from server1 Actual results: =============== glusterd crashed with bt mentioned above, and logs mentioned below: pending frames: frame : type(0) op(0) frame : type(0) op(0) patchset: git://git.gluster.com/glusterfs.git signal received: 11 time of crash: 2014-01-07 00:01:34configuration details: argp 1 backtrace 1 dlfcn 1 fdatasync 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 3.4.0.snap.dec30.2013git /lib64/libc.so.6[0x32130329a0] /lib64/libpthread.so.0(pthread_spin_lock+0x0)[0x321340c380] /usr/lib64/libglusterfs.so.0(mem_put+0x6f)[0x3f09c3f43f] /usr/lib64/libglusterfs.so.0(dict_destroy+0x55)[0x3f09c183c5] /usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_fini_ctx+0x18)[0x7fd2dfa75818] /usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(+0x45801)[0x7fd2dfa7d801] /usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_op_sm+0x1f0)[0x7fd2dfa7a0d0] /usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(__glusterd_handle_commit_op+0x10b)[0x7fd2dfa6445b] /usr/lib64/glusterfs/3.4.0.snap.dec30.2013git/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fd2dfa613cf] /usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x3f09c4cdd2] /lib64/libc.so.6[0x3213043bf0] --------- Expected results: ================= glusterd should not crash
*** Bug 1049166 has been marked as a duplicate of this bug. ***
Fix at http://review.gluster.org/#/c/6728/
Marking snapshot BZs to RHS 3.0.
Setting flags required to add BZs to RHS 3.0 Errata
Version : glusterfs-3.6.0.11-1.el6rhs ======= Retried the steps as mentioned in 'Steps to reproduce'' did not face the issue. Marking the bug as 'Verified'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-1278.html