Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 583811 Details for
Bug 820887
glusterfs process crashed
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Mount log file
mnt-gfsc1.log (text/x-log), 92.29 KB, created by
Shwetha Panduranga
on 2012-05-11 11:37:18 UTC
(
hide
)
Description:
Mount log file
Filename:
MIME Type:
Creator:
Shwetha Panduranga
Created:
2012-05-11 11:37:18 UTC
Size:
92.29 KB
patch
obsolete
>[2012-05-11 05:18:03.231152] I [glusterfsd.c:1629:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.3.0qa40 >[2012-05-11 05:18:04.397683] I [io-cache.c:1558:check_cache_size_ok] 0-vol-quick-read: Max cache size is 2104397824 >[2012-05-11 05:18:04.405687] I [io-cache.c:1558:check_cache_size_ok] 0-vol-io-cache: Max cache size is 2104397824 >[2012-05-11 05:18:04.513474] I [client.c:2151:notify] 0-vol-client-0: parent translators are ready, attempting connect on transport >[2012-05-11 05:18:04.610332] I [client.c:2151:notify] 0-vol-client-1: parent translators are ready, attempting connect on transport >[2012-05-11 05:18:04.703771] I [client.c:2151:notify] 0-vol-client-2: parent translators are ready, attempting connect on transport >[2012-05-11 05:18:04.797745] I [client.c:2151:notify] 0-vol-client-3: parent translators are ready, attempting connect on transport >[2012-05-11 05:18:04.891178] I [client.c:2151:notify] 0-vol-client-4: parent translators are ready, attempting connect on transport >[2012-05-11 05:18:04.985268] I [client.c:2151:notify] 0-vol-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume vol-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.58 > 4: option remote-subvolume /export/brick0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume vol-client-1 > 9: type protocol/client > 10: option remote-host 172.17.251.59 > 11: option remote-subvolume /export/brick1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume vol-client-2 > 16: type protocol/client > 17: option remote-host 172.17.251.60 > 18: option remote-subvolume /export/brick2 > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume vol-client-3 > 23: type protocol/client > 24: option remote-host 172.17.251.58 > 25: option remote-subvolume /export/brick3 > 26: option transport-type tcp > 27: end-volume > 28: > 29: volume vol-client-4 > 30: type protocol/client > 31: option remote-host 172.17.251.59 > 32: option remote-subvolume /export/brick4 > 33: option transport-type tcp > 34: end-volume > 35: > 36: volume vol-client-5 > 37: type protocol/client > 38: option remote-host 172.17.251.60 > 39: option remote-subvolume /export/brick5 > 40: option transport-type tcp > 41: end-volume > 42: > 43: volume vol-replicate-0 > 44: type cluster/replicate > 45: subvolumes vol-client-0 vol-client-1 vol-client-2 > 46: end-volume > 47: > 48: volume vol-replicate-1 > 49: type cluster/replicate > 50: subvolumes vol-client-3 vol-client-4 vol-client-5 > 51: end-volume > 52: > 53: volume vol-dht > 54: type cluster/distribute > 55: subvolumes vol-replicate-0 vol-replicate-1 > 56: end-volume > 57: > 58: volume vol-write-behind > 59: type performance/write-behind > 60: subvolumes vol-dht > 61: end-volume > 62: > 63: volume vol-read-ahead > 64: type performance/read-ahead > 65: subvolumes vol-write-behind > 66: end-volume > 67: > 68: volume vol-io-cache > 69: type performance/io-cache > 70: subvolumes vol-read-ahead > 71: end-volume > 72: > 73: volume vol-quick-read > 74: type performance/quick-read > 75: subvolumes vol-io-cache > 76: end-volume > 77: > 78: volume vol-md-cache > 79: type performance/md-cache > 80: subvolumes vol-quick-read > 81: end-volume > 82: > 83: volume vol > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes vol-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-05-11 05:18:05.127991] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-0: changing port to 24009 (from 0) >[2012-05-11 05:18:05.132838] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-1: changing port to 24009 (from 0) >[2012-05-11 05:18:05.133276] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-2: changing port to 24009 (from 0) >[2012-05-11 05:18:05.133787] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-3: changing port to 24010 (from 0) >[2012-05-11 05:18:05.150178] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-4: changing port to 24010 (from 0) >[2012-05-11 05:18:05.150850] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-vol-client-5: changing port to 24010 (from 0) >[2012-05-11 05:18:08.974565] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.004539] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:18:09.010041] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.023797] I [afr-common.c:3629:afr_notify] 0-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online. >[2012-05-11 05:18:09.035535] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1 >[2012-05-11 05:18:09.116149] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.121610] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'. >[2012-05-11 05:18:09.122014] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.124502] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1 >[2012-05-11 05:18:09.318805] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.325250] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'. >[2012-05-11 05:18:09.325688] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.376796] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-2: Server lk version = 1 >[2012-05-11 05:18:09.608519] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.613876] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:18:09.614098] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.614773] I [afr-common.c:3629:afr_notify] 0-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online. >[2012-05-11 05:18:09.615277] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1 >[2012-05-11 05:18:09.711883] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.713446] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'. >[2012-05-11 05:18:09.713672] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.714592] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1 >[2012-05-11 05:18:09.814541] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:18:09.816277] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'. >[2012-05-11 05:18:09.816518] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:18:09.885727] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0 >[2012-05-11 05:18:09.890548] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-5: Server lk version = 1 >[2012-05-11 05:18:09.903244] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 >[2012-05-11 05:18:10.004028] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 0-vol-replicate-0: added root inode >[2012-05-11 05:18:10.122141] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 0-vol-replicate-1: added root inode >[2012-05-11 05:30:17.603601] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:30:17.609434] I [client.c:2099:client_rpc_notify] 0-vol-client-0: disconnected >[2012-05-11 05:30:22.223944] W [socket.c:204:__socket_rwv] 0-vol-client-3: writev failed (Broken pipe) >[2012-05-11 05:30:22.267448] I [socket.c:1807:socket_event_handler] 0-transport: disconnecting now >[2012-05-11 05:30:22.402461] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.224378 (xid=0x92785x) >[2012-05-11 05:30:22.419607] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.422332] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.231646 (xid=0x92786x) >[2012-05-11 05:30:22.422708] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.423246] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.237831 (xid=0x92787x) >[2012-05-11 05:30:22.423590] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.429332] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.238971 (xid=0x92788x) >[2012-05-11 05:30:22.430776] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.432882] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.242193 (xid=0x92789x) >[2012-05-11 05:30:22.433237] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.433803] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.243210 (xid=0x92790x) >[2012-05-11 05:30:22.434124] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.434667] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.249772 (xid=0x92791x) >[2012-05-11 05:30:22.434977] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.440550] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.254993 (xid=0x92792x) >[2012-05-11 05:30:22.441894] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.443442] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:30:22.256066 (xid=0x92793x) >[2012-05-11 05:30:22.443786] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.444316] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.203825 (xid=0x92783x) >[2012-05-11 05:30:22.445234] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.445869] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.214829 (xid=0x92784x) >[2012-05-11 05:30:22.446183] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.451019] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:22.261213 (xid=0x92794x) >[2012-05-11 05:30:22.455535] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:22.459044] I [client.c:2099:client_rpc_notify] 0-vol-client-3: disconnected >[2012-05-11 05:30:22.463296] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.476787] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.478253] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.483383] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.487423] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.494090] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.495557] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.501414] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.530972] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.544974] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.553846] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.556724] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.557697] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.558897] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.566596] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.570265] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.571465] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:22.578989] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-3: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:28.277359] E [socket.c:1724:socket_connect_finish] 0-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:30:33.124536] E [socket.c:1724:socket_connect_finish] 0-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:30:52.747429] W [socket.c:204:__socket_rwv] 0-vol-client-4: writev failed (Connection reset by peer) >[2012-05-11 05:30:52.752393] I [socket.c:1807:socket_event_handler] 0-transport: disconnecting now >[2012-05-11 05:30:52.753185] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.725643 (xid=0x98227x) >[2012-05-11 05:30:52.757681] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:52.758367] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.728072 (xid=0x98228x) >[2012-05-11 05:30:52.758737] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:52.759280] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.732929 (xid=0x98229x) >[2012-05-11 05:30:52.764772] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:52.765438] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.742158 (xid=0x98230x) >[2012-05-11 05:30:52.765772] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:52.766324] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-4: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:30:52.747726 (xid=0x98231x) >[2012-05-11 05:30:52.770810] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 0-vol-client-4: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:30:52.771111] I [client.c:2099:client_rpc_notify] 0-vol-client-4: disconnected >[2012-05-11 05:30:52.791233] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.798062] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.804841] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.830089] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.837645] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.844871] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.845995] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.853042] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.854133] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.859431] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.867732] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.869020] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:52.907666] W [client3_1-fops.c:5253:client3_1_finodelk] 0-vol-client-4: (44995c04-d34d-4264-b559-cd23d5557ac9) remote_fd is -1. EBADFD >[2012-05-11 05:30:55.113321] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.59:24009) >[2012-05-11 05:30:55.113815] I [client.c:2099:client_rpc_notify] 0-vol-client-1: disconnected >[2012-05-11 05:31:03.975636] E [socket.c:1724:socket_connect_finish] 0-vol-client-4: connection to 172.17.251.59:24010 failed (Connection refused) >[2012-05-11 05:31:05.585948] E [socket.c:1724:socket_connect_finish] 0-vol-client-1: connection to 172.17.251.59:24009 failed (Connection refused) >[2012-05-11 05:34:30.748825] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:34:30.755305] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:34:30.757686] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:34:30.759409] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-0: Server lk version = 1 >[2012-05-11 05:34:32.129841] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:34:32.137772] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'. >[2012-05-11 05:34:32.138094] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:34:32.144126] I [client-handshake.c:1283:client_post_handshake] 0-vol-client-4: 1 fds open - Delaying child_up until they are re-opened >[2012-05-11 05:34:32.206814] I [client-lk.c:610:decrement_reopen_fd_count] 0-vol-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP >[2012-05-11 05:34:32.233428] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-4: Server lk version = 1 >[2012-05-11 05:34:32.302017] I [afr-common.c:1347:afr_launch_self_heal] 0-vol-replicate-1: background data missing-entry gfid self-heal triggered. path: /fuse1.1/dir.1/file.3, reason: subvolume came online >[2012-05-11 05:34:32.379047] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:34:32.414802] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.425423] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.425937] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.426360] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.426785] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.427181] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.427620] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.428027] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.464050] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-4: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3 >[2012-05-11 05:34:32.593230] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-vol-replicate-1: path /fuse1.1/dir.1/file.3 on subvolume vol-client-4 => -1 (No such file or directory) >[2012-05-11 05:34:32.782404] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:34:32.783182] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.783664] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.784073] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.784522] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.784942] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.788684] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.789166] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.789607] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:32.798574] I [afr-open.c:327:afr_openfd_fix_open_cbk] 0-vol-replicate-1: fd for /fuse1.1/dir.1/file.3 opened successfully on subvolume vol-client-4 >[2012-05-11 05:34:33.623453] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:34:33.689286] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:34:33.689650] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:34:33.689848] I [client-handshake.c:1283:client_post_handshake] 0-vol-client-3: 1 fds open - Delaying child_up until they are re-opened >[2012-05-11 05:34:33.703704] I [client-lk.c:610:decrement_reopen_fd_count] 0-vol-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP >[2012-05-11 05:34:33.740093] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-3: Server lk version = 1 >[2012-05-11 05:34:33.998364] I [client-handshake.c:1637:select_server_supported_programs] 0-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:34:34.001735] I [client-handshake.c:1434:client_setvolume_cbk] 0-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'. >[2012-05-11 05:34:34.001968] I [client-handshake.c:1446:client_setvolume_cbk] 0-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:34:34.011245] I [client-handshake.c:462:client_set_lk_version_cbk] 0-vol-client-1: Server lk version = 1 >[2012-05-11 05:34:34.018863] I [afr-common.c:1347:afr_launch_self_heal] 0-vol-replicate-1: background data missing-entry gfid self-heal triggered. path: /fuse1.1/dir.1/file.3, reason: subvolume came online >[2012-05-11 05:34:34.020187] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:34:34.020994] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.021469] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.021899] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.022305] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.022743] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.023160] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.031623] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.032140] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.035065] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-3: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3 >[2012-05-11 05:34:34.116324] E [afr-self-heal-common.c:1011:afr_sh_common_lookup_resp_handler] 0-vol-replicate-1: path /fuse1.1/dir.1/file.3 on subvolume vol-client-3 => -1 (No such file or directory) >[2012-05-11 05:34:34.169808] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:34:34.170677] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.171140] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.171584] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.172003] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.175680] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.176154] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.176603] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.177020] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.187130] W [client3_1-fops.c:419:client3_1_open_cbk] 0-vol-client-3: remote operation failed: No such file or directory. Path: /fuse1.1/dir.1/file.3 >[2012-05-11 05:34:34.326411] I [afr-inode-write.c:437:afr_open_fd_fix] 0-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:34:34.327192] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.327679] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.328084] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.328505] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.328930] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.336358] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.336988] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.337420] I [afr-inode-write.c:415:afr_open_fd_fix] 0-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:34:34.339415] I [afr-open.c:327:afr_openfd_fix_open_cbk] 0-vol-replicate-1: fd for /fuse1.1/dir.1/file.3 opened successfully on subvolume vol-client-3 >[2012-05-11 05:35:02.260508] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-05-11 05:35:03.629567] I [io-cache.c:1558:check_cache_size_ok] 1-vol-quick-read: Max cache size is 2104397824 >[2012-05-11 05:35:03.630562] I [io-cache.c:1558:check_cache_size_ok] 1-vol-io-cache: Max cache size is 2104397824 >[2012-05-11 05:35:03.640219] I [quota.c:3085:quota_parse_limits] 1-vol-quota: no "limit-set" option provided >[2012-05-11 05:36:42.199386] I [client.c:2151:notify] 1-vol-client-0: parent translators are ready, attempting connect on transport >[2012-05-11 05:36:42.299897] I [client.c:2151:notify] 1-vol-client-1: parent translators are ready, attempting connect on transport >[2012-05-11 05:36:42.391690] I [client.c:2151:notify] 1-vol-client-2: parent translators are ready, attempting connect on transport >[2012-05-11 05:36:42.483385] I [client.c:2151:notify] 1-vol-client-3: parent translators are ready, attempting connect on transport >[2012-05-11 05:36:42.575198] I [client.c:2151:notify] 1-vol-client-4: parent translators are ready, attempting connect on transport >[2012-05-11 05:36:42.667066] I [client.c:2151:notify] 1-vol-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume vol-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.58 > 4: option remote-subvolume /export/brick0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume vol-client-1 > 9: type protocol/client > 10: option remote-host 172.17.251.59 > 11: option remote-subvolume /export/brick1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume vol-client-2 > 16: type protocol/client > 17: option remote-host 172.17.251.60 > 18: option remote-subvolume /export/brick2 > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume vol-client-3 > 23: type protocol/client > 24: option remote-host 172.17.251.58 > 25: option remote-subvolume /export/brick3 > 26: option transport-type tcp > 27: end-volume > 28: > 29: volume vol-client-4 > 30: type protocol/client > 31: option remote-host 172.17.251.59 > 32: option remote-subvolume /export/brick4 > 33: option transport-type tcp > 34: end-volume > 35: > 36: volume vol-client-5 > 37: type protocol/client > 38: option remote-host 172.17.251.60 > 39: option remote-subvolume /export/brick5 > 40: option transport-type tcp > 41: end-volume > 42: > 43: volume vol-replicate-0 > 44: type cluster/replicate > 45: subvolumes vol-client-0 vol-client-1 vol-client-2 > 46: end-volume > 47: > 48: volume vol-replicate-1 > 49: type cluster/replicate > 50: subvolumes vol-client-3 vol-client-4 vol-client-5 > 51: end-volume > 52: > 53: volume vol-dht > 54: type cluster/distribute > 55: subvolumes vol-replicate-0 vol-replicate-1 > 56: end-volume > 57: > 58: volume vol-quota > 59: type features/quota > 60: option timeout 0 > 61: subvolumes vol-dht > 62: end-volume > 63: > 64: volume vol-write-behind > 65: type performance/write-behind > 66: subvolumes vol-quota > 67: end-volume > 68: > 69: volume vol-read-ahead > 70: type performance/read-ahead > 71: subvolumes vol-write-behind > 72: end-volume > 73: > 74: volume vol-io-cache > 75: type performance/io-cache > 76: subvolumes vol-read-ahead > 77: end-volume > 78: > 79: volume vol-quick-read > 80: type performance/quick-read > 81: subvolumes vol-io-cache > 82: end-volume > 83: > 84: volume vol-md-cache > 85: type performance/md-cache > 86: subvolumes vol-quick-read > 87: end-volume > 88: > 89: volume vol > 90: type debug/io-stats > 91: option latency-measurement off > 92: option count-fop-hits off > 93: subvolumes vol-md-cache > 94: end-volume > >+------------------------------------------------------------------------------+ >[2012-05-11 05:36:42.763494] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-05-11 05:36:42.770204] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-1: changing port to 24009 (from 0) >[2012-05-11 05:36:42.770732] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-2: changing port to 24009 (from 0) >[2012-05-11 05:36:42.771168] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-4: changing port to 24010 (from 0) >[2012-05-11 05:36:42.772664] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-5: changing port to 24010 (from 0) >[2012-05-11 05:36:42.841122] I [quota.c:3060:quota_parse_limits] 1-vol-quota: /:161061273600 >[2012-05-11 05:36:42.842800] I [quota.c:3092:quota_parse_limits] 1-vol-quota: /:161061273600 >[2012-05-11 05:36:42.854422] I [io-cache.c:1558:check_cache_size_ok] 1-vol-io-cache: Max cache size is 2104397824 >[2012-05-11 05:36:42.856760] I [io-cache.c:1558:check_cache_size_ok] 1-vol-quick-read: Max cache size is 2104397824 >[2012-05-11 05:36:42.863939] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-0: changing port to 24009 (from 0) >[2012-05-11 05:36:42.865932] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 1-vol-client-3: changing port to 24010 (from 0) >[2012-05-11 05:36:46.336653] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:46.338618] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'. >[2012-05-11 05:36:46.338855] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:46.339624] I [afr-common.c:3629:afr_notify] 1-vol-replicate-0: Subvolume 'vol-client-1' came back up; going online. >[2012-05-11 05:36:46.355163] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-1: Server lk version = 1 >[2012-05-11 05:36:46.452497] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:46.454322] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'. >[2012-05-11 05:36:46.454564] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:46.455508] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-2: Server lk version = 1 >[2012-05-11 05:36:46.620304] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:46.627051] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'. >[2012-05-11 05:36:46.627261] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:46.627929] I [afr-common.c:3629:afr_notify] 1-vol-replicate-1: Subvolume 'vol-client-4' came back up; going online. >[2012-05-11 05:36:46.651668] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-4: Server lk version = 1 >[2012-05-11 05:36:46.735320] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:46.737081] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'. >[2012-05-11 05:36:46.737291] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:46.738176] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-5: Server lk version = 1 >[2012-05-11 05:36:46.913675] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:46.917702] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:36:46.917955] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:46.920136] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-0: Server lk version = 1 >[2012-05-11 05:36:47.097094] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:36:47.100719] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:36:47.100939] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:36:47.211667] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 1 >[2012-05-11 05:36:47.213305] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-3: Server lk version = 1 >[2012-05-11 05:38:56.988094] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-05-11 05:38:59.133193] I [io-cache.c:1558:check_cache_size_ok] 2-vol-quick-read: Max cache size is 2104397824 >[2012-05-11 05:38:59.133836] I [io-cache.c:1558:check_cache_size_ok] 2-vol-io-cache: Max cache size is 2104397824 >[2012-05-11 05:38:59.137816] I [quota.c:3060:quota_parse_limits] 2-vol-quota: /:161061273600 >[2012-05-11 05:38:59.138026] I [quota.c:3092:quota_parse_limits] 2-vol-quota: /:161061273600 >[2012-05-11 05:40:27.615613] I [client.c:2151:notify] 2-vol-client-0: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:27.714936] I [client.c:2151:notify] 2-vol-client-1: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:27.815179] I [client.c:2151:notify] 2-vol-client-2: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:27.982869] I [client.c:2151:notify] 2-vol-client-3: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:28.184909] I [client.c:2151:notify] 2-vol-client-4: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:28.380038] I [client.c:2151:notify] 2-vol-client-5: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:28.580583] I [client.c:2151:notify] 2-vol-client-6: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:28.781464] I [client.c:2151:notify] 2-vol-client-7: parent translators are ready, attempting connect on transport >[2012-05-11 05:40:28.981042] I [client.c:2151:notify] 2-vol-client-8: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume vol-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.58 > 4: option remote-subvolume /export/brick0 > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume vol-client-1 > 9: type protocol/client > 10: option remote-host 172.17.251.59 > 11: option remote-subvolume /export/brick1 > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume vol-client-2 > 16: type protocol/client > 17: option remote-host 172.17.251.60 > 18: option remote-subvolume /export/brick2 > 19: option transport-type tcp > 20: end-volume > 21: > 22: volume vol-client-3 > 23: type protocol/client > 24: option remote-host 172.17.251.58 > 25: option remote-subvolume /export/brick3 > 26: option transport-type tcp > 27: end-volume > 28: > 29: volume vol-client-4 > 30: type protocol/client > 31: option remote-host 172.17.251.59 > 32: option remote-subvolume /export/brick4 > 33: option transport-type tcp > 34: end-volume > 35: > 36: volume vol-client-5 > 37: type protocol/client > 38: option remote-host 172.17.251.60 > 39: option remote-subvolume /export/brick5 > 40: option transport-type tcp > 41: end-volume > 42: > 43: volume vol-client-6 > 44: type protocol/client > 45: option remote-host 172.17.251.58 > 46: option remote-subvolume /export/brick6 > 47: option transport-type tcp > 48: end-volume > 49: > 50: volume vol-client-7 > 51: type protocol/client > 52: option remote-host 172.17.251.59 > 53: option remote-subvolume /export/brick7 > 54: option transport-type tcp > 55: end-volume > 56: > 57: volume vol-client-8 > 58: type protocol/client > 59: option remote-host 172.17.251.59 > 60: option remote-subvolume /export/brick8 > 61: option transport-type tcp > 62: end-volume > 63: > 64: volume vol-replicate-0 > 65: type cluster/replicate > 66: subvolumes vol-client-0 vol-client-1 vol-client-2 > 67: end-volume > 68: > 69: volume vol-replicate-1 > 70: type cluster/replicate > 71: subvolumes vol-client-3 vol-client-4 vol-client-5 > 72: end-volume > 73: > 74: volume vol-replicate-2 > 75: type cluster/replicate > 76: subvolumes vol-client-6 vol-client-7 vol-client-8 > 77: end-volume > 78: > 79: volume vol-dht > 80: type cluster/distribute > 81: subvolumes vol-replicate-0 vol-replicate-1 vol-replicate-2 > 82: end-volume > 83: > 84: volume vol-quota > 85: type features/quota > 86: option limit-set /:150GB > 87: option timeout 0 > 88: subvolumes vol-dht > 89: end-volume > 90: > 91: volume vol-write-behind > 92: type performance/write-behind > 93: subvolumes vol-quota > 94: end-volume > 95: > 96: volume vol-read-ahead > 97: type performance/read-ahead > 98: subvolumes vol-write-behind > 99: end-volume >100: >101: volume vol-io-cache >102: type performance/io-cache >103: subvolumes vol-read-ahead >104: end-volume >105: >106: volume vol-quick-read >107: type performance/quick-read >108: subvolumes vol-io-cache >109: end-volume >110: >111: volume vol-md-cache >112: type performance/md-cache >113: subvolumes vol-quick-read >114: end-volume >115: >116: volume vol >117: type debug/io-stats >118: option latency-measurement off >119: option count-fop-hits off >120: subvolumes vol-md-cache >121: end-volume > >+------------------------------------------------------------------------------+ >[2012-05-11 05:40:29.203205] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-0: changing port to 24009 (from 0) >[2012-05-11 05:40:29.203822] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-1: changing port to 24009 (from 0) >[2012-05-11 05:40:29.204330] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-3: changing port to 24010 (from 0) >[2012-05-11 05:40:29.212082] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-7: changing port to 24011 (from 0) >[2012-05-11 05:40:29.212639] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-6: changing port to 24011 (from 0) >[2012-05-11 05:40:29.213107] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-4: changing port to 24010 (from 0) >[2012-05-11 05:40:29.213621] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-2: changing port to 24009 (from 0) >[2012-05-11 05:40:29.214487] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-8: changing port to 24012 (from 0) >[2012-05-11 05:40:29.214939] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 2-vol-client-5: changing port to 24010 (from 0) >[2012-05-11 05:40:33.162000] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:33.171726] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:40:33.172211] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:33.173633] I [afr-common.c:3629:afr_notify] 2-vol-replicate-0: Subvolume 'vol-client-0' came back up; going online. >[2012-05-11 05:40:33.187651] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-0: Server lk version = 1 >[2012-05-11 05:40:33.645702] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-1: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:33.656032] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-1: Connected to 172.17.251.59:24009, attached to remote volume '/export/brick1'. >[2012-05-11 05:40:33.656259] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:33.657437] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-1: Server lk version = 1 >[2012-05-11 05:40:33.999599] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:34.001392] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:40:34.001620] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:34.002301] I [afr-common.c:3629:afr_notify] 2-vol-replicate-1: Subvolume 'vol-client-3' came back up; going online. >[2012-05-11 05:40:34.002842] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-3: Server lk version = 1 >[2012-05-11 05:40:34.039362] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:40:34.039742] I [client.c:2099:client_rpc_notify] 0-vol-client-0: disconnected >[2012-05-11 05:40:34.048753] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:40:34.049082] I [client.c:2099:client_rpc_notify] 1-vol-client-0: disconnected >[2012-05-11 05:40:34.049434] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:40:34.049824] I [client.c:2099:client_rpc_notify] 2-vol-client-0: disconnected >[2012-05-11 05:40:34.384127] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-7: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:34.395302] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-7: Connected to 172.17.251.59:24011, attached to remote volume '/export/brick7'. >[2012-05-11 05:40:34.395558] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:34.396222] I [afr-common.c:3629:afr_notify] 2-vol-replicate-2: Subvolume 'vol-client-7' came back up; going online. >[2012-05-11 05:40:34.396744] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-7: Server lk version = 1 >[2012-05-11 05:40:34.730378] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-6: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:34.732434] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-6: Connected to 172.17.251.58:24011, attached to remote volume '/export/brick6'. >[2012-05-11 05:40:34.732655] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:34.740633] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-6: Server lk version = 1 >[2012-05-11 05:40:35.346168] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-4: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:35.348116] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-4: Connected to 172.17.251.59:24010, attached to remote volume '/export/brick4'. >[2012-05-11 05:40:35.348401] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:35.361146] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-4: Server lk version = 1 >[2012-05-11 05:40:35.680868] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-2: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:35.683000] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-2: Connected to 172.17.251.60:24009, attached to remote volume '/export/brick2'. >[2012-05-11 05:40:35.683302] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:35.695776] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-2: Server lk version = 1 >[2012-05-11 05:40:36.015691] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-8: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:36.020227] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-8: Connected to 172.17.251.59:24012, attached to remote volume '/export/brick8'. >[2012-05-11 05:40:36.020466] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:36.022503] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-8: Server lk version = 1 >[2012-05-11 05:40:36.330214] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-5: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:40:36.332765] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-5: Connected to 172.17.251.60:24010, attached to remote volume '/export/brick5'. >[2012-05-11 05:40:36.332994] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:40:36.704658] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 2 >[2012-05-11 05:40:36.709974] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-5: Server lk version = 1 >[2012-05-11 05:40:37.384844] W [socket.c:1521:__socket_proto_state_machine] 0-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010) >[2012-05-11 05:40:37.393448] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.794282 (xid=0x92902x) >[2012-05-11 05:40:37.412549] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.422720] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.795266 (xid=0x92903x) >[2012-05-11 05:40:37.439785] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.443432] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.796214 (xid=0x92904x) >[2012-05-11 05:40:37.454146] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.461193] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.797145 (xid=0x92905x) >[2012-05-11 05:40:37.469234] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.471228] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.811409 (xid=0x92906x) >[2012-05-11 05:40:37.471898] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.482026] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.812499 (xid=0x92907x) >[2012-05-11 05:40:37.482764] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.484315] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.813305 (xid=0x92908x) >[2012-05-11 05:40:37.492129] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.493855] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.813941 (xid=0x92909x) >[2012-05-11 05:40:37.494545] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.500277] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 0-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:34:34.814570 (xid=0x92910x) >[2012-05-11 05:40:37.506618] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:40:37.507778] I [client.c:2099:client_rpc_notify] 0-vol-client-3: disconnected >[2012-05-11 05:40:37.513508] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010) >[2012-05-11 05:40:37.514237] I [client.c:2099:client_rpc_notify] 2-vol-client-3: disconnected >[2012-05-11 05:40:37.514847] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010) >[2012-05-11 05:40:37.520475] I [client.c:2099:client_rpc_notify] 1-vol-client-3: disconnected >[2012-05-11 05:40:37.963768] E [socket.c:1724:socket_connect_finish] 2-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:40:41.305768] E [socket.c:1724:socket_connect_finish] 2-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:40:45.365632] E [socket.c:1724:socket_connect_finish] 0-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:40:45.715449] E [socket.c:1724:socket_connect_finish] 1-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:40:48.458590] E [socket.c:1724:socket_connect_finish] 0-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:40:48.658551] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-6: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24011) >[2012-05-11 05:40:48.658979] I [client.c:2099:client_rpc_notify] 2-vol-client-6: disconnected >[2012-05-11 05:40:48.735946] E [socket.c:1724:socket_connect_finish] 1-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:40:59.725666] E [socket.c:1724:socket_connect_finish] 2-vol-client-6: connection to 172.17.251.58:24011 failed (Connection refused) >[2012-05-11 05:41:50.440620] I [afr-self-heal-algorithm.c:131:sh_loop_driver_done] 0-vol-replicate-1: diff self-heal on /fuse1.1/dir.1/file.3: completed. (0 blocks of 5901 were different (0.00%)) >[2012-05-11 05:41:50.556018] W [client3_1-fops.c:4876:client3_1_fxattrop] 0-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:41:50.556971] E [afr-self-heal-data.c:1000:afr_sh_common_fxattrop_resp_handler] 0-vol-replicate-1: fxattrop of /fuse1.1/dir.1/file.3 failed on vol-client-3, reason File descriptor in bad state >[2012-05-11 05:41:50.565323] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 0-vol-replicate-1: background data missing-entry gfid self-heal completed on /fuse1.1/dir.1/file.3 >[2012-05-11 05:41:50.592745] W [client3_1-fops.c:3135:client3_1_ftruncate] 0-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:41:50.595233] E [afr-self-heal-data.c:604:afr_sh_data_trim_cbk] 0-vol-replicate-1: ftruncate of /fuse1.1/dir.1/file.3 on subvolume vol-client-3 failed (File descriptor in bad state) >[2012-05-11 05:41:50.598043] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 0-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:41:50.606027] W [client3_1-fops.c:4049:client3_1_flush] 0-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:41:50.608173] I [afr-self-heal-data.c:115:afr_sh_data_flush_cbk] 0-vol-replicate-1: flush failed on /fuse1.1/dir.1/file.3 on subvolume vol-client-3: File descriptor in bad state >[2012-05-11 05:41:50.629880] E [afr-self-heal-common.c:2051:afr_self_heal_completion_cbk] 0-vol-replicate-1: background data missing-entry gfid self-heal failed on /fuse1.1/dir.1/file.3 >[2012-05-11 05:41:50.656965] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-1: added root inode >[2012-05-11 05:41:50.675040] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-0: added root inode >[2012-05-11 05:41:50.678886] I [afr-common.c:1971:afr_set_root_inode_on_first_lookup] 2-vol-replicate-2: added root inode >[2012-05-11 05:41:51.226959] I [afr-common.c:1347:afr_launch_self_heal] 2-vol-replicate-1: background meta-data data self-heal triggered. path: , reason: lookup detected pending operations >[2012-05-11 05:41:51.790044] I [afr-self-heal-data.c:781:afr_sh_data_fix] 2-vol-replicate-1: no active sinks for performing self-heal on file >[2012-05-11 05:41:51.827289] I [afr-self-heal-common.c:2054:afr_self_heal_completion_cbk] 2-vol-replicate-1: background meta-data data self-heal completed on >[2012-05-11 05:41:51.923854] I [client.c:2160:notify] 0-vol-client-0: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.928028] I [client.c:2160:notify] 0-vol-client-1: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.930333] I [client.c:2160:notify] 0-vol-client-2: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.930709] I [client.c:2160:notify] 0-vol-client-3: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.930943] I [client.c:2160:notify] 0-vol-client-4: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.931607] I [client.c:2160:notify] 0-vol-client-5: current graph is no longer active, destroying rpc_client >[2012-05-11 05:41:51.932522] I [client.c:2099:client_rpc_notify] 0-vol-client-1: disconnected >[2012-05-11 05:41:51.932971] I [client.c:2099:client_rpc_notify] 0-vol-client-2: disconnected >[2012-05-11 05:41:51.933466] E [afr-common.c:3666:afr_notify] 0-vol-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-05-11 05:41:51.934680] I [client.c:2099:client_rpc_notify] 0-vol-client-4: disconnected >[2012-05-11 05:41:51.935027] I [client.c:2099:client_rpc_notify] 0-vol-client-5: disconnected >[2012-05-11 05:41:51.935238] E [afr-common.c:3666:afr_notify] 0-vol-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-05-11 05:42:26.774298] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:42:26.789596] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:42:26.789937] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:42:26.792135] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-0: Server lk version = 1 >[2012-05-11 05:42:27.986857] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:42:27.990788] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:42:27.991069] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:42:27.998980] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-3: Server lk version = 1 >[2012-05-11 05:42:28.103124] I [afr-common.c:1347:afr_launch_self_heal] 2-vol-replicate-1: background data missing-entry gfid self-heal triggered. path: <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94>, reason: subvolume came online >[2012-05-11 05:42:28.123597] I [afr-inode-write.c:437:afr_open_fd_fix] 2-vol-replicate-1: Opening fd 0xc09c6dc >[2012-05-11 05:42:28.124704] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.125241] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.125819] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.126326] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.131901] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.132455] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.132967] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.133480] I [afr-inode-write.c:415:afr_open_fd_fix] 2-vol-replicate-1: Pause fd 0xc09c6dc >[2012-05-11 05:42:28.144418] I [afr-open.c:327:afr_openfd_fix_open_cbk] 2-vol-replicate-1: fd for <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> opened successfully on subvolume vol-client-3 >[2012-05-11 05:42:28.176184] I [client-handshake.c:1637:select_server_supported_programs] 2-vol-client-6: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:42:28.183778] I [client-handshake.c:1434:client_setvolume_cbk] 2-vol-client-6: Connected to 172.17.251.58:24011, attached to remote volume '/export/brick6'. >[2012-05-11 05:42:28.184072] I [client-handshake.c:1446:client_setvolume_cbk] 2-vol-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:42:28.186269] I [client-handshake.c:462:client_set_lk_version_cbk] 2-vol-client-6: Server lk version = 1 >[2012-05-11 05:42:29.493204] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-3: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:42:29.503086] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-3: Connected to 172.17.251.58:24010, attached to remote volume '/export/brick3'. >[2012-05-11 05:42:29.503422] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:42:29.506106] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-3: Server lk version = 1 >[2012-05-11 05:42:29.670397] I [client-handshake.c:1637:select_server_supported_programs] 1-vol-client-0: Using Program GlusterFS 3.3.0qa40, Num (1298437), Version (330) >[2012-05-11 05:42:29.674741] I [client-handshake.c:1434:client_setvolume_cbk] 1-vol-client-0: Connected to 172.17.251.58:24009, attached to remote volume '/export/brick0'. >[2012-05-11 05:42:29.674976] I [client-handshake.c:1446:client_setvolume_cbk] 1-vol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-05-11 05:42:29.677481] I [client-handshake.c:462:client_set_lk_version_cbk] 1-vol-client-0: Server lk version = 1 >[2012-05-11 05:43:35.972544] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:43:35.972974] I [client.c:2099:client_rpc_notify] 2-vol-client-0: disconnected >[2012-05-11 05:43:35.973333] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24009) >[2012-05-11 05:43:35.973672] I [client.c:2099:client_rpc_notify] 1-vol-client-0: disconnected >[2012-05-11 05:43:41.314055] W [socket.c:204:__socket_rwv] 2-vol-client-3: writev failed (Connection reset by peer) >[2012-05-11 05:43:41.327078] I [socket.c:1807:socket_event_handler] 2-transport: disconnecting now >[2012-05-11 05:43:41.328173] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(INODELK(29)) called at 2012-05-11 05:43:41.294053 (xid=0x40001x) >[2012-05-11 05:43:41.328565] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.329265] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.316731 (xid=0x40011x) >[2012-05-11 05:43:41.329616] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.330170] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.317886 (xid=0x40012x) >[2012-05-11 05:43:41.330519] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.331062] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.319010 (xid=0x40013x) >[2012-05-11 05:43:41.331403] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.331954] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.320109 (xid=0x40014x) >[2012-05-11 05:43:41.332277] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.332863] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.321235 (xid=0x40015x) >[2012-05-11 05:43:41.333188] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.333757] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.322385 (xid=0x40016x) >[2012-05-11 05:43:41.334092] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.334649] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.323505 (xid=0x40017x) >[2012-05-11 05:43:41.334985] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.335546] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.324607 (xid=0x40018x) >[2012-05-11 05:43:41.335877] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.336441] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-05-11 05:43:41.325749 (xid=0x40019x) >[2012-05-11 05:43:41.336780] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.337317] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(RCHECKSUM(37)) called at 2012-05-11 05:43:41.294963 (xid=0x40002x) >[2012-05-11 05:43:41.338321] W [client3_1-fops.c:2099:client3_1_rchecksum_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.373978] E [afr-self-heal-algorithm.c:618:sh_diff_checksum_cbk] 2-vol-replicate-1: checksum on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> failed on subvolume vol-client-3 (Transport endpoint is not connected) >[2012-05-11 05:43:41.471816] I [socket.c:2324:socket_submit_request] 2-vol-client-3: not connected (priv->connected = 0) >[2012-05-11 05:43:41.472855] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40021x Program: GlusterFS 3.1, ProgVers: 330, Proc: 29) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.473526] W [client3_1-fops.c:1493:client3_1_inodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.475926] I [afr-self-heal-algorithm.c:118:sh_loop_driver_done] 2-vol-replicate-1: self-heal aborting on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> >[2012-05-11 05:43:41.476795] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40022x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.477660] W [client3_1-fops.c:882:client3_1_flush_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.477908] I [afr-self-heal-data.c:115:afr_sh_data_flush_cbk] 2-vol-replicate-1: flush failed on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> on subvolume vol-client-3: Transport endpoint is not connected >[2012-05-11 05:43:41.479437] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.297481 (xid=0x40003x) >[2012-05-11 05:43:41.479805] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.480207] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40023x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.480487] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.481682] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.299476 (xid=0x40004x) >[2012-05-11 05:43:41.482006] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.482385] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40024x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.482635] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.483742] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.302331 (xid=0x40005x) >[2012-05-11 05:43:41.484112] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.484492] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40025x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.484751] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.485897] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.304391 (xid=0x40006x) >[2012-05-11 05:43:41.486212] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.486594] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40026x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.486850] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.488000] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.306735 (xid=0x40007x) >[2012-05-11 05:43:41.488314] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.488726] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40027x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.488976] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.490088] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.308905 (xid=0x40008x) >[2012-05-11 05:43:41.490443] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.490811] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40028x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.491062] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.492192] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.310851 (xid=0x40009x) >[2012-05-11 05:43:41.492539] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.492917] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40029x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.493171] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.494308] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.314395 (xid=0x40010x) >[2012-05-11 05:43:41.494667] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.495024] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40030x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.495277] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.496421] E [rpc-clnt.c:382:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x13c) [0x4eaec4e] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x155) [0x4eae16d] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0x1f) [0x4eadbb3]))) 2-vol-client-3: forced unwinding frame type(GlusterFS 3.1) op(FXATTROP(34)) called at 2012-05-11 05:43:41.326780 (xid=0x40020x) >[2012-05-11 05:43:41.496750] W [client3_1-fops.c:1777:client3_1_fxattrop_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.497107] W [rpc-clnt.c:1507:rpc_clnt_submit] 2-vol-client-3: failed to submit rpc-request (XID: 0x40031x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (vol-client-3) >[2012-05-11 05:43:41.497377] W [client3_1-fops.c:1543:client3_1_finodelk_cbk] 2-vol-client-3: remote operation failed: Transport endpoint is not connected >[2012-05-11 05:43:41.498188] I [client.c:2099:client_rpc_notify] 2-vol-client-3: disconnected >[2012-05-11 05:43:41.498526] W [socket.c:1521:__socket_proto_state_machine] 1-vol-client-3: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24010) >[2012-05-11 05:43:41.498864] I [client.c:2099:client_rpc_notify] 1-vol-client-3: disconnected >[2012-05-11 05:43:41.500121] E [socket.c:1724:socket_connect_finish] 2-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:43:41.508464] E [afr-self-heal-common.c:2051:afr_self_heal_completion_cbk] 2-vol-replicate-1: background data self-heal failed on <gfid:7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94> >[2012-05-11 05:43:41.514761] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.515825] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.516760] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.517693] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.518605] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.519536] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.520478] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.521409] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:41.522325] W [client3_1-fops.c:5253:client3_1_finodelk] 2-vol-client-3: (7a5f6d08-a098-4eb4-8ea5-4cc5d95d5b94) remote_fd is -1. EBADFD >[2012-05-11 05:43:44.426634] W [socket.c:1521:__socket_proto_state_machine] 2-vol-client-6: reading from socket failed. Error (Transport endpoint is not connected), peer (172.17.251.58:24011) >[2012-05-11 05:43:44.427060] I [client.c:2099:client_rpc_notify] 2-vol-client-6: disconnected >[2012-05-11 05:43:46.890773] E [socket.c:1724:socket_connect_finish] 2-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:43:47.101102] E [socket.c:1724:socket_connect_finish] 1-vol-client-0: connection to 172.17.251.58:24009 failed (Connection refused) >[2012-05-11 05:43:52.888026] E [socket.c:1724:socket_connect_finish] 1-vol-client-3: connection to 172.17.251.58:24010 failed (Connection refused) >[2012-05-11 05:43:55.600167] E [socket.c:1724:socket_connect_finish] 2-vol-client-6: connection to 172.17.251.58:24011 failed (Connection refused) >pending frames: >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) >frame : type(1) op(WRITE) > >patchset: git://git.gluster.com/glusterfs.git >signal received: 11 >time of crash: 2012-05-11 05:44:08 >configuration details: >argp 1 >backtrace 1 >dlfcn 1 >fdatasync 1 >libpthread 1 >llistxattr 1 >setfsid 1 >spinlock 1 >epoll.h 1 >xattr.h 1 >st_atim.tv_nsec 1 >package-string: glusterfs 3.3.0qa40 >/lib64/libc.so.6[0x3a89632900] >/usr/local/lib/libglusterfs.so.0(fd_ctx_dump+0x151)[0x4c57258] >/usr/local/lib/libglusterfs.so.0(inode_dump+0x1f8)[0x4c41e6a] >/usr/local/lib/libglusterfs.so.0(inode_table_dump+0x2ac)[0x4c42237] >/usr/local/lib/libglusterfs.so.0(gf_proc_dump_xlator_info+0x100)[0x4c61457] >/usr/local/lib/libglusterfs.so.0(gf_proc_dump_info+0x239)[0x4c61ec0] >/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0x11a)[0x407973] >/lib64/libpthread.so.0[0x3a89a077f1] >/lib64/libc.so.6(clone+0x6d)[0x3a896e5ccd] >---------
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 820887
: 583811 |
583812
|
583813