Description of problem:
Replicate volume with replica count 2. 1 nfs client and 1 fuse client. volume set operations were running in a loop. On nfs client rdd was running and on fuse sanity script. glusterfs server crashed with the below backtrace

Core was generated by `/usr/local/sbin/glusterfsd -s localhost --volfile-id mirror.10.1.11.172.export-'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f1c332386ff in grant_blocked_inode_locks (this=0x6f3ea0, pl_inode=0x7f1c2400c9c0, dom=0x7f1c24004370)
    at ../../../../../xlators/features/locks/src/inodelk.c:365
365                     STACK_UNWIND_STRICT (inodelk, lock->frame, 0, 0);
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.25.el6.x86_64 libgcc-4.4.5-6.el6.x86_64
(gdb) bt
#0  0x00007f1c332386ff in grant_blocked_inode_locks (this=0x6f3ea0, pl_inode=0x7f1c2400c9c0, dom=0x7f1c24004370)
    at ../../../../../xlators/features/locks/src/inodelk.c:365
#1  0x00007f1c33239053 in pl_inode_setlk (this=0x6f3ea0, pl_inode=0x7f1c2400c9c0, lock=0x7f1c24004590, can_block=0, dom=0x7f1c24004370)
    at ../../../../../xlators/features/locks/src/inodelk.c:506
#2  0x00007f1c332395aa in pl_common_inodelk (frame=0x7f1c3614bfe0, this=0x6f3ea0, volume=0xe565a0 "mirror-replicate-0", 
    inode=0x7f1c31bc5a20, cmd=6, flock=0x7f1c35e2130c, loc=0x7f1c35e212c4, fd=0x0) at ../../../../../xlators/features/locks/src/inodelk.c:618
#3  0x00007f1c332398fe in pl_inodelk (frame=0x7f1c3614bfe0, this=0x6f3ea0, volume=0xe565a0 "mirror-replicate-0", loc=0x7f1c35e212c4, cmd=6, 
    flock=0x7f1c35e2130c) at ../../../../../xlators/features/locks/src/inodelk.c:661
#4  0x00007f1c3301e6b9 in iot_inodelk_wrapper (frame=0x7f1c361501b8, this=0x6f5030, volume=0xe565a0 "mirror-replicate-0", 
    loc=0x7f1c35e212c4, cmd=6, lock=0x7f1c35e2130c) at ../../../../../xlators/performance/io-threads/src/io-threads.c:1939
#5  0x00007f1c3753c381 in call_resume_wind (stub=0x7f1c35e21284) at ../../../libglusterfs/src/call-stub.c:2360
#6  0x00007f1c37543446 in call_resume (stub=0x7f1c35e21284) at ../../../libglusterfs/src/call-stub.c:3853
#7  0x00007f1c3301179d in iot_worker (data=0x6ffad0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:138
#8  0x00000034a44077e1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000034a40e68ed in clone () from /lib64/libc.so.6
(gdb) f 0
#0  0x00007f1c332386ff in grant_blocked_inode_locks (this=0x6f3ea0, pl_inode=0x7f1c2400c9c0, dom=0x7f1c24004370)
    at ../../../../../xlators/features/locks/src/inodelk.c:365
365                     STACK_UNWIND_STRICT (inodelk, lock->frame, 0, 0);
(gdb) l
360                             lock->user_flock.l_len);
361
362                     pl_trace_out (this, lock->frame, NULL, NULL, F_SETLKW,
363                                   &lock->user_flock, 0, 0, lock->volume);
364
365                     STACK_UNWIND_STRICT (inodelk, lock->frame, 0, 0);
366             }
367
368     }
369
(gdb) p lock->frame
$1 = (call_frame_t *) 0x65
(gdb) p *lock
$2 = {list = {next = 0x7f1c1000bc50, prev = 0x7f1c10023120}, blocked_locks = {next = 0x7f1c1000b700, prev = 0x0}, fl_type = 0, fl_start = 0, 
  fl_end = 0, volume = 0x0, user_flock = {l_type = 0, l_whence = 0, l_start = 0, l_len = 0, l_pid = 831636728, l_owner = 139668028251056}, 
  this = 0xff35dbb0, fd = 0xbaadf00d, frame = 0x65, blkd_time = {tv_sec = 81604378664, tv_usec = 31398341237538816}, granted_time = {
    tv_sec = -3819410108757049344, tv_usec = 0}, transport = 0x722d726f7272696d, client_pid = 1768714341, owner = 52545592190054445}
(gdb) info thr
  9 Thread 0x7f1c3181a700 (LWP 26768)  0xffffffffff60042d in ?? ()
  8 Thread 0x7f1c3191b700 (LWP 26766)  0x00000034a440e074 in __lll_lock_wait () from /lib64/libpthread.so.0
  7 Thread 0x7f1c3541b700 (LWP 26717)  0x00000034a440b44c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 Thread 0x7f1c35e1c700 (LWP 26716)  0x00000034a440f2c5 in sigwait () from /lib64/libpthread.so.0
  5 Thread 0x7f1c370bd700 (LWP 26715)  0x00000034a40e6ee3 in epoll_wait () from /lib64/libc.so.6
  4 Thread 0x7f1c325eb700 (LWP 26730)  0x00000034a440b7cb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 Thread 0x7f1c23fff700 (LWP 26767)  0x00000034a440a64b in pthread_mutex_unlock () from /lib64/libpthread.so.0
  2 Thread 0x7f1c3427a700 (LWP 26719)  0x00000034a440ed4d in nanosleep () from /lib64/libpthread.so.0
* 1 Thread 0x7f1c326ec700 (LWP 26729)  0x00007f1c332386ff in grant_blocked_inode_locks (this=0x6f3ea0, pl_inode=0x7f1c2400c9c0, 
    dom=0x7f1c24004370) at ../../../../../xlators/features/locks/src/inodelk.c:365
(gdb) t 9
[Switching to thread 9 (Thread 0x7f1c3181a700 (LWP 26768))]#0  0xffffffffff60042d in ?? ()
(gdb) bt
#0  0xffffffffff60042d in ?? ()
#1  0x000000004f18f50f in ?? ()
#2  0x00000000000bcc1b in ?? ()
#3  0x00007f1c31819e7c in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) t 8
[Switching to thread 8 (Thread 0x7f1c3191b700 (LWP 26766))]#0  0x00000034a440e074 in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000034a440e074 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000034a4409345 in _L_lock_870 () from /lib64/libpthread.so.0
#2  0x00000034a4409217 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f1c33011646 in iot_worker (data=0x6ffad0) at ../../../../../xlators/performance/io-threads/src/io-threads.c:101
#4  0x00000034a44077e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000034a40e68ed in clone () from /lib64/libc.so.6
(gdb) 
 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

log information:

it/xlator/protocol/server.so(server_finodelk_cbk+0x24b) [0x7f1c329ae7d9]))) 0-: Reply submission failed
[2012-01-20 00:01:03.711881] I [server-helpers.c:328:do_lock_table_cleanup] 0-mirror-server: finodelk released on /okpa
[2012-01-20 00:01:03.711910] E [rpcsvc.c:1060:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x16x, Program: GlusterFS 3git, ProgVers: 310, Proc: 30) to rpc-transport (tcp.mirror-server)
[2012-01-20 00:01:03.711953] E [server.c:142:server_submit_reply] (-->/usr/local/lib/libglusterfs.so.0(default_finodelk_cbk+0x14d) [0x7f1c3751d0e8] (-->/usr/local/lib/glusterfs/3git/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0x23a) [0x7f1c32bdb622] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/server.so(server_finodelk_cbk+0x24b) [0x7f1c329ae7d9]))) 0-: Reply submission failed
[2012-01-20 00:01:03.711972] I [server-helpers.c:328:do_lock_table_cleanup] 0-mirror-server: finodelk released on /okpa
[2012-01-20 00:01:03.711998] E [rpcsvc.c:1060:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x14x, Program: GlusterFS 3git, ProgVers: 310, Proc: 30) to rpc-transport (tcp.mirror-server)
[2012-01-20 00:01:03.712085] E [server.c:142:server_submit_reply] (-->/usr/local/lib/libglusterfs.so.0(default_finodelk_cbk+0x14d) [0x7f1c3751d0e8] (-->/usr/local/lib/glusterfs/3git/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0x23a) [0x7f1c32bdb622] (-->/usr/local/lib/glusterfs/3git/xlator/protocol/server.so(server_finodelk_cbk+0x24b) [0x7f1c329ae7d9]))) 0-: Reply submission failed
[2012-01-20 00:01:03.712179] I [server-helpers.c:328:do_lock_table_cleanup] 0-mirror-server: finodelk released on /okpa
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2012-01-20 00:01:03
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3git
/lib64/libc.so.6[0x34a4032ac0]
/usr/local/lib/glusterfs/3git/xlator/features/locks.so(grant_blocked_inode_locks+0x26b)[0x7f1c332386ff]
/usr/local/lib/glusterfs/3git/xlator/features/locks.so(+0x12053)[0x7f1c33239053]
/usr/local/lib/glusterfs/3git/xlator/features/locks.so(pl_common_inodelk+0x429)[0x7f1c332395aa]
/usr/local/lib/glusterfs/3git/xlator/features/locks.so(pl_inodelk+0x5e)[0x7f1c332398fe]
/usr/local/lib/glusterfs/3git/xlator/performance/io-threads.so(iot_inodelk_wrapper+0x248)[0x7f1c3301e6b9]
/usr/local/lib/libglusterfs.so.0(+0x3e381)[0x7f1c3753c381]
/usr/local/lib/libglusterfs.so.0(call_resume+0xd2)[0x7f1c37543446]
/usr/local/lib/glusterfs/3git/xlator/performance/io-threads.so(iot_worker+0x1f3)[0x7f1c3301179d]
/lib64/libpthread.so.0[0x34a44077e1]
/lib64/libc.so.6(clone+0x6d)[0x34a40e68ed]