when dbench is run with 4 servers ( 1 of the servers has error-gen ) and distribute on the client side, the client crashes


Client log:

[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28034: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28037: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28042: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:959:dht_attr_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:959:dht_attr_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28055: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:959:dht_attr_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:959:dht_attr_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:959:dht_attr_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28120: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28122: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28125: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:02:38] D [dht-common.c:1278:dht_err_cbk] distribute: subvolume client1 returned -1 (File descriptor in bad state)
[2009-11-11 06:02:38] W [fuse-bridge.c:1191:fuse_err_cbk] glusterfs-fuse: 28128: FLUSH() ERR => -1 (File descriptor in bad state)
[2009-11-11 06:03:12] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)
[2009-11-11 06:03:15] D [dht-common.c:1567:dht_fd_cbk] distribute: subvolume client1 returned -1 (Transport endpoint is not connected)

(gdb) bt
#0  server_fsync_cbk (frame=0x635ec0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, prebuf=<value optimized out>, postbuf=0x6)
    at ../../../../libglusterfs/src/protocol.h:87
#1  0x00002aaaaace11a4 in default_fsync_cbk (frame=0x62a620, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, prebuf=0x7fff338feab0, postbuf=0x6)
    at defaults.c:630
#2  0x00002aaaac12678e in error_gen_fsync_cbk (frame=0x6273f0, cookie=<value optimized out>, this=<value optimized out>, op_ret=0, op_errno=0, prebuf=<value optimized out>,
    postbuf=0x7fff338fea20) at error-gen.c:1333
#3  0x00002aaaabf151f9 in posix_fsync (frame=0x621df0, this=<value optimized out>, fd=<value optimized out>, datasync=<value optimized out>) at posix.c:2752
#4  0x00002aaaac12add0 in error_gen_fsync (frame=0x6273f0, this=0x60e990, fd=0x61c690, flags=0) at error-gen.c:1361
#5  0x00002aaaaace10ee in default_fsync (frame=<value optimized out>, this=0x60f440, fd=0x61c690, flags=0) at defaults.c:644
#6  0x00002aaaac552854 in server_fsync_resume (frame=<value optimized out>, bound_xl=0x60f440) at server-protocol.c:2876
#7  0x00002aaaac554c62 in server_resolve_done (frame=0x4afa99ce) at server-resolve.c:515
#8  0x00002aaaac55539f in server_resolve_all (frame=0x4afa99ce) at server-resolve.c:550
#9  0x00002aaaac5552c6 in server_resolve (frame=0x4afa99ce) at server-resolve.c:499
#10 0x00002aaaac555398 in server_resolve_all (frame=0x4afa99ce) at server-resolve.c:546
#11 0x00002aaaac55548c in server_resolve_fd (frame=0x635ec0) at server-resolve.c:465
#12 0x00002aaaac55529e in server_resolve (frame=0x4afa99ce) at server-resolve.c:484
#13 0x00002aaaac555378 in server_resolve_all (frame=0x4afa99ce) at server-resolve.c:539
#14 0x00002aaaac555433 in resolve_and_resume (frame=0x635ec0, fn=0x4a) at server-resolve.c:574
#15 0x00002aaaac547654 in server_fsync (frame=0x635ec0, bound_xl=<value optimized out>, hdr=<value optimized out>, hdrlen=<value optimized out>, iobuf=<value optimized out>)
    at server-protocol.c:2903
#16 0x00002aaaac545430 in protocol_server_pollin (this=0x60fd70, trans=0x612a20) at server-protocol.c:6361
#17 0x00002aaaac54550b in notify (this=0x60fd70, event=<value optimized out>, data=0xce00) at server-protocol.c:6417
#18 0x00002aaaaacdaea3 in xlator_notify (xl=0x60fd70, event=2, data=0x612a20) at xlator.c:923
#19 0x00002aaaac762f23 in socket_event_handler (fd=<value optimized out>, idx=1, data=0x612a20, poll_in=1, poll_out=0, poll_err=0) at socket.c:829
#20 0x00002aaaaacf4205 in event_dispatch_epoll (event_pool=0x608330) at event.c:804
#21 0x000000000040407a in main (argc=5, argv=0x7fff338ffb88) at glusterfsd.c:1379