Description of problem:


Version-Release number of selected component (if applicable):
Mainline

How reproducible:


Steps to Reproduce:
1. create a distributed-stripe-replicate volume 2x2x2
2. run sanity

  
Actual results:
glusterfs crashed

Expected results:


Additional info:
Core was generated by `/usr/local/sbin/glusterfs --volfile-id=dist-str-rep --volfile-server=10.1.11.14'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ff5b4bef65e in stripe_writev (frame=0x7ff5b851f5f4, this=0x130c4e0, fd=0x1461be0, vector=0x7ff5a42d2fe0, count=1, offset=3117, flags=33793,
    iobref=0x7ff5a4175180) at stripe.c:3768
3768                    STACK_WIND (frame, stripe_writev_cbk, fctx->xl_array[idx],



===========================================================================

(gdb) p fctx->xl_array[idx]
$3 = (xlator_t *) 0x0


===========================================================
#0  0x00007ff5b4bef65e in stripe_writev (frame=0x7ff5b851f5f4, this=0x130c4e0, fd=0x1461be0, vector=0x7ff5a42d2fe0, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a4175180) at stripe.c:3768
#1  0x00007ff5b49869c4 in dht_writev (frame=0x7ff5b851e528, this=0x130d150, fd=0x1461be0, vector=0x7ff5a42d2fe0, count=1, off=3117, flags=33793, 
    iobref=0x7ff5a4175180) at dht-inode-write.c:157
#2  0x00007ff5b470d925 in wb_sync (frame=0x7ff5b830c0b8, file=0x2790f30, winds=0x7ff5ac09a690) at write-behind.c:547
#3  0x00007ff5b4717de5 in wb_do_ops (frame=0x7ff5b830c0b8, file=0x2790f30, winds=0x7ff5ac09a690, unwinds=0x7ff5ac09a680, other_requests=0x7ff5ac09a670)
    at write-behind.c:1888
#4  0x00007ff5b4718bc0 in wb_process_queue (frame=0x7ff5b830c0b8, file=0x2790f30) at write-behind.c:2078
#5  0x00007ff5b4719b73 in wb_writev (frame=0x7ff5b8520c20, this=0x130e4b0, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a41a6550) at write-behind.c:2199
#6  0x00007ff5b44f9687 in ra_writev (frame=0x7ff5b851ae10, this=0x130f7a0, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a41a6550) at read-ahead.c:688
#7  0x00007ff5b42dab83 in ioc_writev (frame=0x7ff5b851f3f0, this=0x13109f0, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a41a6550) at io-cache.c:1248
#8  0x00007ff5b40ae628 in qr_writev (frame=0x7ff5b8518bcc, this=0x1311c40, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, off=3117, wr_flags=33793, 
    iobref=0x7ff5a41a6550) at quick-read.c:1466
#9  0x00007ff5afdf3c05 in mdc_writev (frame=0x7ff5b851f5f4, this=0x1312fc0, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a41a6550) at md-cache.c:1329
#10 0x00007ff5afbd54d0 in io_stats_writev (frame=0x7ff5b851e528, this=0x1314250, fd=0x1461be0, vector=0x7ff5a42cff88, count=1, offset=3117, flags=33793, 
    iobref=0x7ff5a41a6550) at io-stats.c:2091
#11 0x00007ff5b7fadc9d in fuse_write_resume (state=0x7ff5a42cf890) at fuse-bridge.c:2033
#12 0x00007ff5b7f9a47f in fuse_resolve_done (state=0x7ff5a42cf890) at fuse-resolve.c:453
#13 0x00007ff5b7f9a5c4 in fuse_resolve_all (state=0x7ff5a42cf890) at fuse-resolve.c:482
#14 0x00007ff5b7f9a424 in fuse_resolve (state=0x7ff5a42cf890) at fuse-resolve.c:439
#15 0x00007ff5b7f9a574 in fuse_resolve_all (state=0x7ff5a42cf890) at fuse-resolve.c:478
#16 0x00007ff5b7f9a6bf in fuse_resolve_continue (state=0x7ff5a42cf890) at fuse-resolve.c:498
#17 0x00007ff5b7f9a00b in fuse_resolve_fd (state=0x7ff5a42cf890) at fuse-resolve.c:351
#18 0x00007ff5b7f9a33c in fuse_resolve (state=0x7ff5a42cf890) at fuse-resolve.c:428
#19 0x00007ff5b7f9a4fb in fuse_resolve_all (state=0x7ff5a42cf890) at fuse-resolve.c:471
#20 0x00007ff5b7f9a733 in fuse_resolve_and_resume (state=0x7ff5a42cf890, fn=0x7ff5b7fad432 <fuse_write_resume>) at fuse-resolve.c:511
#21 0x00007ff5b7fadfda in fuse_write (this=0x12ea750, finh=0x7ff5a42d18a0, msg=0x7ff5b8ae0000) at fuse-bridge.c:2079
#22 0x00007ff5b7fbd204 in fuse_thread_proc (data=0x12ea750) at fuse-bridge.c:3976
#23 0x000000390fc077e1 in start_thread () from /lib64/libpthread.so.0
#24 0x000000390f4e577d in clone () from /lib64/libc.so.6
======================================================================


[2012-03-15 08:20:09.707847] W [client.c:1992:client_rpc_notify] 0-dist-str-rep-client-6: Cancelling the grace timer
[2012-03-15 08:20:09.707915] I [client-handshake.c:1533:select_server_supported_programs] 0-dist-str-rep-client-6: Using Program GlusterFS 3git, Num (1298437), Version
 (330)
[2012-03-15 08:20:09.708142] I [client-handshake.c:1308:client_setvolume_cbk] 0-dist-str-rep-client-6: clnt-lk-version = 1, server-lk-version = 0
[2012-03-15 08:20:09.708184] I [client-handshake.c:1334:client_setvolume_cbk] 0-dist-str-rep-client-6: Connected to 10.1.11.141:24015, attached to remote volume '/export/sda/dist-str-rep7'.
[2012-03-15 08:20:09.708254] I [afr-common.c:3484:afr_notify] 0-dist-str-rep-replicate-3: Subvolume 'dist-str-rep-client-6' came back up; going online.
[2012-03-15 08:20:09.710725] W [client.c:1992:client_rpc_notify] 0-dist-str-rep-client-7: Cancelling the grace timer
[2012-03-15 08:20:09.710802] I [client-handshake.c:1533:select_server_supported_programs] 0-dist-str-rep-client-7: Using Program GlusterFS 3git, Num (1298437), Version (330)
[2012-03-15 08:20:09.711126] I [client-handshake.c:1308:client_setvolume_cbk] 0-dist-str-rep-client-7: clnt-lk-version = 1, server-lk-version = 0
[2012-03-15 08:20:09.711154] I [client-handshake.c:1334:client_setvolume_cbk] 0-dist-str-rep-client-7: Connected to 10.1.11.141:24016, attached to remote volume '/export/sda/dist-str-rep8'.
[2012-03-15 08:20:09.719411] I [fuse-bridge.c:4095:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-03-15 08:20:09.719681] I [fuse-bridge.c:3342:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-03-15 08:20:09.722022] I [afr-common.c:1850:afr_set_root_inode_on_first_lookup] 0-dist-str-rep-replicate-0: added root inode
[2012-03-15 08:20:09.722560] I [afr-common.c:1850:afr_set_root_inode_on_first_lookup] 0-dist-str-rep-replicate-1: added root inode
[2012-03-15 08:20:09.723991] I [afr-common.c:1850:afr_set_root_inode_on_first_lookup] 0-dist-str-rep-replicate-2: added root inode
[2012-03-15 08:20:09.724045] I [afr-common.c:1850:afr_set_root_inode_on_first_lookup] 0-dist-str-rep-replicate-3: added root inode
[2012-03-15 11:36:41.186854] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-7: remote operation failed: No such file or directory
[2012-03-15 11:36:41.236426] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-6: remote operation failed: No such file or directory
[2012-03-15 11:36:41.298652] I [afr-common.c:1313:afr_launch_self_heal] 0-dist-str-rep-replicate-3: background  entry self-heal triggered. path: /run4296/glusterfs.git/.git, reason: lookup detected pending operations
[2012-03-15 11:36:41.299418] I [afr-self-heal-entry.c:2332:afr_sh_entry_fix] 0-dist-str-rep-replicate-3: /run4296/glusterfs.git/.git: Performing conservative merge
[2012-03-15 11:36:41.306289] I [afr-self-heal-common.c:2037:afr_self_heal_completion_cbk] 0-dist-str-rep-replicate-3: background  entry self-heal completed on /run4296/glusterfs.git/.git
[2012-03-15 11:41:15.487967] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-2: remote operation failed: No such file or directory
[2012-03-15 11:41:15.581383] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-3: remote operation failed: No such file or directory
[2012-03-15 11:41:15.632066] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-6: remote operation failed: No such file or directory
[2012-03-15 11:41:15.632129] W [client3_1-fops.c:511:client3_1_unlink_cbk] 0-dist-str-rep-client-7: remote operation failed: No such file or directory
[2012-03-15 11:41:25.270999] I [afr-common.c:1313:afr_launch_self_heal] 0-dist-str-rep-replicate-1: background  entry self-heal triggered. path: /run4296/glusterfs.git, reason: lookup detected pending operations
[2012-03-15 11:41:25.271984] I [afr-self-heal-entry.c:2332:afr_sh_entry_fix] 0-dist-str-rep-replicate-1: /run4296/glusterfs.git: Performing conservative merge
[2012-03-15 11:41:25.897456] I [afr-self-heal-common.c:2037:afr_self_heal_completion_cbk] 0-dist-str-rep-replicate-1: background  entry self-heal completed on /run4296