Created attachment 576225 [details]
Backtrace of core

Description of problem:
Observed glusterd crash when restarted. 

(gdb) bt
#0  0x000000363872f9df in __strlen_sse42 () from /lib64/libc.so.6
#1  0x00007f35b00a0098 in gf_strdup (src=0x241 <Address 0x241 out of bounds>) at mem-pool.h:128
#2  0x00007f35b00a037a in runner_add_arg (runner=0x7f35ab33ad90, arg=0x241 <Address 0x241 out of bounds>) at run.c:117
#3  0x00007f35b00a03cf in runner_va_add_args (runner=0x7f35ab33ad90, argp=0x7f35ab334840) at run.c:132
#4  0x00007f35b00a04eb in runner_add_args (runner=0x7f35ab33ad90) at run.c:141
#5  0x00007f35ac5a25d1 in glusterd_volume_start_glusterfs (volinfo=0xebefb0, brickinfo=0xec4bf0) at glusterd-utils.c:1230
#6  0x00007f35ac5a972e in glusterd_brick_start (volinfo=0xebefb0, brickinfo=0xec4bf0) at glusterd-utils.c:3332
#7  0x00007f35ac5a98a9 in glusterd_brick_restart_proc (data=0xeb5630) at glusterd-utils.c:3364
#8  0x0000003638a077f1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000036386e570d in clone () from /lib64/libc.so.6

[04/10/12 - 01:06:24 root@APP-SERVER2 ~]# gluster volume info
 
Volume Name: dstore
Type: Distributed-Replicate
Volume ID: 6212e353-88cd-4cf2-846e-34f07d11d14b
Status: Started
Number of Bricks: 5 x 3 = 15
Transport-type: tcp
Bricks:
Brick1: 192.168.2.35:/export1/dstore1/1
Brick2: 192.168.2.36:/export1/dstore1/1
Brick3: 192.168.2.37:/export1/dstore1/1
Brick4: 192.168.2.35:/export1/dstore1/2
Brick5: 192.168.2.36:/export1/dstore1/2
Brick6: 192.168.2.37:/export1/dstore1/2
Brick7: 192.168.2.35:/export1/dstore1/3
Brick8: 192.168.2.36:/export1/dstore1/3
Brick9: 192.168.2.37:/export1/dstore1/3
Brick10: 192.168.2.35:/export1/dstore1/4
Brick11: 192.168.2.36:/export1/dstore1/4
Brick12: 192.168.2.37:/export1/dstore1/4
Brick13: 192.168.2.35:/export1/dstore1/5
Brick14: 192.168.2.36:/export1/dstore1/5
Brick15: 192.168.2.37:/export1/dstore1/5
Options Reconfigured:
cluster.self-heal-daemon: off
features.lock-heal: off
server.grace-timeout: 10
client.grace-timeout: 10


Version-Release number of selected component (if applicable):
mainline

gfsc1.sh
--------
#!/bin/bash

mountpoint=`pwd`
for i in {1..10}
do
	level1_dir=$mountpoint/fuse2.$i
	mkdir $level1_dir
	cd $level1_dir
	for j in {1..20}
	do 
		level2_dir=dir.$j
		mkdir $level2_dir
		cd $level2_dir
		for k in {1..100}
		do 
			echo "Creating File: $leve1_dir/$level2_dir/file.$k"
			dd if=/dev/zero of=file.$k bs=1M count=$k 
		done
		cd $level1_dir
	done
	cd $mountpoint
done

Steps to Reproduce:
1.create a distribute-replicate volume(5X3). (3 machines in trusted storage pool ). start the volume
2.create a fuse mount and run gfsc1.sh script from mount
3.bounce bricks from machine1. 
4.trigger heal on the volume. (volume heal cli command)
5.restart machine1.
6.turn off self-heal daemon on the volume. 
7.set client/server.grace-timeout to 10.
8.start glusterd on machine1. 
  
Actual results:
glusterd crashed

Additional Info:-
Machine1 :- 192.168.2.35
Machine2 :- 192.168.2.36
Machine3 :- 192.168.2.37

volume:- dstore
core:- /core.1883

Machine is in the same state as the state of crash.