Attachment 859502 Details for Bug 1061580 – Fuse mount log files from Hypervisor

Fuse mount log files from Hypervisor

rhev-data-center-mnt-10.70.37.86:dr-store.log (text/x-log), 56.09 KB, created by SATHEESARAN on 2014-02-05 07:17:30 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: SATHEESARAN

Created: 2014-02-05 07:17:30 UTC

Size: 56.09 KB

patch

obsolete

>[2014-02-03 19:17:15.792197] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.58rhs (/usr/sbin/glusterfs --volfile-server=10.70.37.86 --volfile-id=dr-store /rhev/data-center/mnt/10.70.37.86:dr-store)
>[2014-02-03 19:17:15.844962] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-03 19:17:15.845010] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-03 19:17:15.859030] I [dht-shared.c:311:dht_init_regex] 0-dr-store-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-03 19:17:15.862796] I [socket.c:3505:socket_init] 0-dr-store-client-3: SSL support is NOT enabled
>[2014-02-03 19:17:15.862824] I [socket.c:3520:socket_init] 0-dr-store-client-3: using system polling thread
>[2014-02-03 19:17:15.863468] I [socket.c:3505:socket_init] 0-dr-store-client-2: SSL support is NOT enabled
>[2014-02-03 19:17:15.863479] I [socket.c:3520:socket_init] 0-dr-store-client-2: using system polling thread
>[2014-02-03 19:17:15.864116] I [socket.c:3505:socket_init] 0-dr-store-client-1: SSL support is NOT enabled
>[2014-02-03 19:17:15.864128] I [socket.c:3520:socket_init] 0-dr-store-client-1: using system polling thread
>[2014-02-03 19:17:15.864762] I [socket.c:3505:socket_init] 0-dr-store-client-0: SSL support is NOT enabled
>[2014-02-03 19:17:15.864774] I [socket.c:3520:socket_init] 0-dr-store-client-0: using system polling thread
>[2014-02-03 19:17:15.864809] I [client.c:2171:notify] 0-dr-store-client-0: parent translators are ready, attempting connect on transport
>[2014-02-03 19:17:15.867737] I [client.c:2171:notify] 0-dr-store-client-1: parent translators are ready, attempting connect on transport
>[2014-02-03 19:17:15.870558] I [client.c:2171:notify] 0-dr-store-client-2: parent translators are ready, attempting connect on transport
>[2014-02-03 19:17:15.873375] I [client.c:2171:notify] 0-dr-store-client-3: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dr-store-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.86
>  4:     option remote-subvolume /rhs/brick1/drs1
>  5:     option transport-type socket
>  6:     option filter-O_DIRECT enable
>  7: end-volume
>  8: 
>  9: volume dr-store-client-1
> 10:     type protocol/client
> 11:     option remote-host 10.70.37.187
> 12:     option remote-subvolume /rhs/brick1/dsr1
> 13:     option transport-type socket
> 14:     option filter-O_DIRECT enable
> 15: end-volume
> 16: 
> 17: volume dr-store-replicate-0
> 18:     type cluster/replicate
> 19:     option eager-lock enable
> 20:     option quorum-type auto
> 21:     subvolumes dr-store-client-0 dr-store-client-1
> 22: end-volume
> 23: 
> 24: volume dr-store-client-2
> 25:     type protocol/client
> 26:     option remote-host 10.70.37.86
> 27:     option remote-subvolume /rhs/brick2/dsr2
> 28:     option transport-type socket
> 29:     option filter-O_DIRECT enable
> 30: end-volume
> 31: 
> 32: volume dr-store-client-3
> 33:     type protocol/client
> 34:     option remote-host 10.70.37.187
> 35:     option remote-subvolume /rhs/brick2/dsr2
> 36:     option transport-type socket
> 37:     option filter-O_DIRECT enable
> 38: end-volume
> 39: 
> 40: volume dr-store-replicate-1
> 41:     type cluster/replicate
> 42:     option eager-lock enable
> 43:     option quorum-type auto
> 44:     subvolumes dr-store-client-2 dr-store-client-3
> 45: end-volume
> 46: 
> 47: volume dr-store-dht
> 48:     type cluster/distribute
> 49:     subvolumes dr-store-replicate-0 dr-store-replicate-1
> 50: end-volume
> 51: 
> 52: volume dr-store-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes dr-store-dht
> 55: end-volume
> 56: 
> 57: volume dr-store
> 58:     type debug/io-stats
> 59:     option latency-measurement off
> 60:     option count-fop-hits off
> 61:     subvolumes dr-store-write-behind
> 62: end-volume
> 63: 
>+------------------------------------------------------------------------------+
>[2014-02-03 19:17:15.877004] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-dr-store-client-0: changing port to 49152 (from 0)
>[2014-02-03 19:17:15.877049] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-dr-store-client-2: changing port to 49153 (from 0)
>[2014-02-03 19:17:15.883200] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-dr-store-client-1: changing port to 49152 (from 0)
>[2014-02-03 19:17:15.886087] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-dr-store-client-3: changing port to 49153 (from 0)
>[2014-02-03 19:17:15.889027] I [client-handshake.c:1695:select_server_supported_programs] 0-dr-store-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-03 19:17:15.889385] I [client-handshake.c:1695:select_server_supported_programs] 0-dr-store-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-03 19:17:15.889470] I [client-handshake.c:1695:select_server_supported_programs] 0-dr-store-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-03 19:17:15.889530] I [client-handshake.c:1695:select_server_supported_programs] 0-dr-store-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-03 19:17:15.889591] I [client-handshake.c:1466:client_setvolume_cbk] 0-dr-store-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-03 19:17:15.889604] I [client-handshake.c:1480:client_setvolume_cbk] 0-dr-store-client-0: Connected to 10.70.37.86:49152, attached to remote volume '/rhs/brick1/drs1'.
>[2014-02-03 19:17:15.889612] I [client-handshake.c:1492:client_setvolume_cbk] 0-dr-store-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-03 19:17:15.889654] I [afr-common.c:3988:afr_notify] 0-dr-store-replicate-0: Subvolume 'dr-store-client-0' came back up; going online.
>[2014-02-03 19:17:15.889912] I [client-handshake.c:1466:client_setvolume_cbk] 0-dr-store-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-03 19:17:15.889927] I [client-handshake.c:1480:client_setvolume_cbk] 0-dr-store-client-3: Connected to 10.70.37.187:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-03 19:17:15.889934] I [client-handshake.c:1492:client_setvolume_cbk] 0-dr-store-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-03 19:17:15.889992] I [afr-common.c:3988:afr_notify] 0-dr-store-replicate-1: Subvolume 'dr-store-client-3' came back up; going online.
>[2014-02-03 19:17:15.890022] I [client-handshake.c:1466:client_setvolume_cbk] 0-dr-store-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-03 19:17:15.890031] I [client-handshake.c:1480:client_setvolume_cbk] 0-dr-store-client-1: Connected to 10.70.37.187:49152, attached to remote volume '/rhs/brick1/dsr1'.
>[2014-02-03 19:17:15.890049] I [client-handshake.c:1492:client_setvolume_cbk] 0-dr-store-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-03 19:17:15.890165] I [client-handshake.c:450:client_set_lk_version_cbk] 0-dr-store-client-3: Server lk version = 1
>[2014-02-03 19:17:15.890195] I [client-handshake.c:450:client_set_lk_version_cbk] 0-dr-store-client-0: Server lk version = 1
>[2014-02-03 19:17:15.890217] I [client-handshake.c:1466:client_setvolume_cbk] 0-dr-store-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-03 19:17:15.890226] I [client-handshake.c:1480:client_setvolume_cbk] 0-dr-store-client-2: Connected to 10.70.37.86:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-03 19:17:15.890233] I [client-handshake.c:1492:client_setvolume_cbk] 0-dr-store-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-03 19:17:15.894130] I [fuse-bridge.c:5874:fuse_graph_setup] 0-fuse: switched to graph 0
>[2014-02-03 19:17:15.894230] I [client-handshake.c:450:client_set_lk_version_cbk] 0-dr-store-client-2: Server lk version = 1
>[2014-02-03 19:17:15.894252] I [client-handshake.c:450:client_set_lk_version_cbk] 0-dr-store-client-1: Server lk version = 1
>[2014-02-03 19:17:15.894328] I [fuse-bridge.c:4811:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2014-02-03 19:18:41.815047] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-03 19:18:41.815749] I [glusterfsd-mgmt.c:1379:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-02-03 19:18:52.376074] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-03 19:18:52.416655] I [glusterfsd-mgmt.c:1379:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-02-03 19:19:03.058742] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-03 19:19:03.107033] I [glusterfsd-mgmt.c:1379:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-02-03 19:19:05.978998] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-03 19:19:06.029162] I [glusterfsd-mgmt.c:1379:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-02-04 21:00:18.947653] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-04 21:00:18.984992] I [dht-shared.c:311:dht_init_regex] 1-dr-store-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-04 21:00:18.989796] I [socket.c:3505:socket_init] 1-dr-store-client-5: SSL support is NOT enabled
>[2014-02-04 21:00:18.989831] I [socket.c:3520:socket_init] 1-dr-store-client-5: using system polling thread
>[2014-02-04 21:00:18.990465] I [socket.c:3505:socket_init] 1-dr-store-client-4: SSL support is NOT enabled
>[2014-02-04 21:00:18.990477] I [socket.c:3520:socket_init] 1-dr-store-client-4: using system polling thread
>[2014-02-04 21:00:18.991115] I [socket.c:3505:socket_init] 1-dr-store-client-3: SSL support is NOT enabled
>[2014-02-04 21:00:18.991127] I [socket.c:3520:socket_init] 1-dr-store-client-3: using system polling thread
>[2014-02-04 21:00:18.991762] I [socket.c:3505:socket_init] 1-dr-store-client-2: SSL support is NOT enabled
>[2014-02-04 21:00:18.991774] I [socket.c:3520:socket_init] 1-dr-store-client-2: using system polling thread
>[2014-02-04 21:00:18.992405] I [socket.c:3505:socket_init] 1-dr-store-client-1: SSL support is NOT enabled
>[2014-02-04 21:00:18.992417] I [socket.c:3520:socket_init] 1-dr-store-client-1: using system polling thread
>[2014-02-04 21:00:18.993055] I [socket.c:3505:socket_init] 1-dr-store-client-0: SSL support is NOT enabled
>[2014-02-04 21:00:18.993068] I [socket.c:3520:socket_init] 1-dr-store-client-0: using system polling thread
>[2014-02-04 21:00:18.993096] I [client.c:2171:notify] 1-dr-store-client-0: parent translators are ready, attempting connect on transport
>[2014-02-04 21:00:19.000019] I [client.c:2171:notify] 1-dr-store-client-1: parent translators are ready, attempting connect on transport
>[2014-02-04 21:00:19.002824] I [client.c:2171:notify] 1-dr-store-client-2: parent translators are ready, attempting connect on transport
>[2014-02-04 21:00:19.005607] I [client.c:2171:notify] 1-dr-store-client-3: parent translators are ready, attempting connect on transport
>[2014-02-04 21:00:19.008381] I [client.c:2171:notify] 1-dr-store-client-4: parent translators are ready, attempting connect on transport
>[2014-02-04 21:00:19.011144] I [client.c:2171:notify] 1-dr-store-client-5: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dr-store-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.86
>  4:     option remote-subvolume /rhs/brick1/drs1
>  5:     option transport-type socket
>  6:     option filter-O_DIRECT enable
>  7: end-volume
>  8: 
>  9: volume dr-store-client-1
> 10:     type protocol/client
> 11:     option remote-host 10.70.37.187
> 12:     option remote-subvolume /rhs/brick1/dsr1
> 13:     option transport-type socket
> 14:     option filter-O_DIRECT enable
> 15: end-volume
> 16: 
> 17: volume dr-store-replicate-0
> 18:     type cluster/replicate
> 19:     option eager-lock enable
> 20:     option quorum-type auto
> 21:     subvolumes dr-store-client-0 dr-store-client-1
> 22: end-volume
> 23: 
> 24: volume dr-store-client-2
> 25:     type protocol/client
> 26:     option remote-host 10.70.37.86
> 27:     option remote-subvolume /rhs/brick2/dsr2
> 28:     option transport-type socket
> 29:     option filter-O_DIRECT enable
> 30: end-volume
> 31: 
> 32: volume dr-store-client-3
> 33:     type protocol/client
> 34:     option remote-host 10.70.37.187
> 35:     option remote-subvolume /rhs/brick2/dsr2
> 36:     option transport-type socket
> 37:     option filter-O_DIRECT enable
> 38: end-volume
> 39: 
> 40: volume dr-store-replicate-1
> 41:     type cluster/replicate
> 42:     option eager-lock enable
> 43:     option quorum-type auto
> 44:     subvolumes dr-store-client-2 dr-store-client-3
> 45: end-volume
> 46: 
> 47: volume dr-store-client-4
> 48:     type protocol/client
> 49:     option remote-host 10.70.37.86
> 50:     option remote-subvolume /rhs/brick3/dsr3
> 51:     option transport-type socket
> 52:     option filter-O_DIRECT enable
> 53: end-volume
> 54: 
> 55: volume dr-store-client-5
> 56:     type protocol/client
> 57:     option remote-host 10.70.37.187
> 58:     option remote-subvolume /rhs/brick3/dsr3
> 59:     option transport-type socket
> 60:     option filter-O_DIRECT enable
> 61: end-volume
> 62: 
> 63: volume dr-store-replicate-2
> 64:     type cluster/replicate
> 65:     option eager-lock enable
> 66:     option quorum-type auto
> 67:     subvolumes dr-store-client-4 dr-store-client-5
> 68: end-volume
> 69: 
> 70: volume dr-store-dht
> 71:     type cluster/distribute
> 72:     subvolumes dr-store-replicate-0 dr-store-replicate-1 dr-store-replicate-2
> 73: end-volume
> 74: 
> 75: volume dr-store-write-behind
> 76:     type performance/write-behind
> 77:     subvolumes dr-store-dht
> 78: end-volume
> 79: 
> 80: volume dr-store
> 81:     type debug/io-stats
> 82:     option latency-measurement off
> 83:     option count-fop-hits off
> 84:     subvolumes dr-store-write-behind
> 85: end-volume
> 86: 
>+------------------------------------------------------------------------------+
>[2014-02-04 21:00:19.015275] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-3: changing port to 49153 (from 0)
>[2014-02-04 21:00:19.020313] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-1: changing port to 49152 (from 0)
>[2014-02-04 21:00:19.020358] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:00:19.023415] I [client.c:2103:client_rpc_notify] 1-dr-store-client-5: disconnected from 10.70.37.187:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.023791] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:19.023890] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:19.047708] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-2: changing port to 49153 (from 0)
>[2014-02-04 21:00:19.050625] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-0: changing port to 49152 (from 0)
>[2014-02-04 21:00:19.053474] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:00:19.053512] I [client.c:2103:client_rpc_notify] 1-dr-store-client-4: disconnected from 10.70.37.86:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.053526] E [afr-common.c:4025:afr_notify] 1-dr-store-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:00:19.053813] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:19.053994] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:19.054724] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:19.054740] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-3: Connected to 10.70.37.187:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-04 21:00:19.054748] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:19.054791] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-1: Subvolume 'dr-store-client-3' came back up; going online.
>[2014-02-04 21:00:19.054821] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:19.054831] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-1: Connected to 10.70.37.187:49152, attached to remote volume '/rhs/brick1/dsr1'.
>[2014-02-04 21:00:19.054838] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:19.054871] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-0: Subvolume 'dr-store-client-1' came back up; going online.
>[2014-02-04 21:00:19.055086] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-3: Server lk version = 1
>[2014-02-04 21:00:19.055139] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-1: Server lk version = 1
>[2014-02-04 21:00:19.080513] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:19.080529] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-0: Connected to 10.70.37.86:49152, attached to remote volume '/rhs/brick1/drs1'.
>[2014-02-04 21:00:19.080536] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:19.080885] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:19.080907] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-2: Connected to 10.70.37.86:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-04 21:00:19.080916] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:19.084795] I [fuse-bridge.c:5874:fuse_graph_setup] 0-fuse: switched to graph 1
>[2014-02-04 21:00:19.084852] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-2: Server lk version = 1
>[2014-02-04 21:00:19.084929] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-0: Server lk version = 1
>[2014-02-04 21:00:19.088881] I [fuse-bridge.c:5086:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x766e28) to newfd (0x8a762c) (inode-gfid:9189e31b-8388-439a-8e50-2943a31d11cd)(old-subvolume:dr-store-0 new-subvolume:dr-store-1)
>[2014-02-04 21:00:19.091744] I [fuse-bridge.c:5086:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x766e94) to newfd (0x8a7698) (inode-gfid:2287b21f-dc5e-47f5-8730-679ee9b44273)(old-subvolume:dr-store-0 new-subvolume:dr-store-1)
>[2014-02-04 21:00:19.113623] I [fuse-bridge.c:5086:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x766f00) to newfd (0x8a7704) (inode-gfid:3b9018ca-1c74-483d-97c2-f90e85c7bccc)(old-subvolume:dr-store-0 new-subvolume:dr-store-1)
>[2014-02-04 21:00:19.116886] I [fuse-bridge.c:5086:fuse_migrate_fd_open] 0-glusterfs-fuse: migrated basefd (0x766f6c) to newfd (0x8a7770) (inode-gfid:b31d8277-fae7-48c7-a377-1b8fc817a6eb)(old-subvolume:dr-store-0 new-subvolume:dr-store-1)
>[2014-02-04 21:00:19.117031] I [client.c:2180:notify] 0-dr-store-client-0: current graph is no longer active, destroying rpc_client 
>[2014-02-04 21:00:19.117074] I [client.c:2180:notify] 0-dr-store-client-1: current graph is no longer active, destroying rpc_client 
>[2014-02-04 21:00:19.117095] I [client.c:2103:client_rpc_notify] 0-dr-store-client-0: disconnected from 10.70.37.86:49152. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.117101] I [client.c:2180:notify] 0-dr-store-client-2: current graph is no longer active, destroying rpc_client 
>[2014-02-04 21:00:19.117126] I [client.c:2103:client_rpc_notify] 0-dr-store-client-1: disconnected from 10.70.37.187:49152. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.117131] I [client.c:2180:notify] 0-dr-store-client-3: current graph is no longer active, destroying rpc_client 
>[2014-02-04 21:00:19.117143] E [afr-common.c:4025:afr_notify] 0-dr-store-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:00:19.117169] I [client.c:2103:client_rpc_notify] 0-dr-store-client-2: disconnected from 10.70.37.86:49153. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.117186] I [client.c:2103:client_rpc_notify] 0-dr-store-client-3: disconnected from 10.70.37.187:49153. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:00:19.117195] E [afr-common.c:4025:afr_notify] 0-dr-store-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:00:22.213593] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-4: changing port to 49155 (from 0)
>[2014-02-04 21:00:22.219522] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:22.219931] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:22.219946] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-4: Connected to 10.70.37.86:49155, attached to remote volume '/rhs/brick3/dsr3'.
>[2014-02-04 21:00:22.219955] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:22.219991] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-2: Subvolume 'dr-store-client-4' came back up; going online.
>[2014-02-04 21:00:22.220178] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-4: Server lk version = 1
>[2014-02-04 21:00:25.232253] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-5: changing port to 49155 (from 0)
>[2014-02-04 21:00:25.235581] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:00:25.235852] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:00:25.235868] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-5: Connected to 10.70.37.187:49155, attached to remote volume '/rhs/brick3/dsr3'.
>[2014-02-04 21:00:25.235876] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:00:25.236086] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-5: Server lk version = 1
>[2014-02-04 21:35:21.505434] W [socket.c:522:__socket_rwv] 1-dr-store-client-2: readv on 10.70.37.86:49153 failed (No data available)
>[2014-02-04 21:35:21.505512] I [client.c:2103:client_rpc_notify] 1-dr-store-client-2: disconnected from 10.70.37.86:49153. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:21.505551] W [socket.c:522:__socket_rwv] 1-dr-store-client-4: readv on 10.70.37.86:49155 failed (No data available)
>[2014-02-04 21:35:21.505574] I [client.c:2103:client_rpc_notify] 1-dr-store-client-4: disconnected from 10.70.37.86:49155. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:21.505617] W [socket.c:522:__socket_rwv] 1-dr-store-client-0: readv on 10.70.37.86:49152 failed (No data available)
>[2014-02-04 21:35:21.505658] I [client.c:2103:client_rpc_notify] 1-dr-store-client-0: disconnected from 10.70.37.86:49152. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:31.556190] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:35:31.556255] I [client.c:2103:client_rpc_notify] 1-dr-store-client-2: disconnected from 10.70.37.86:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:31.558700] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:35:31.558773] I [client.c:2103:client_rpc_notify] 1-dr-store-client-4: disconnected from 10.70.37.86:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:31.561213] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:35:31.561263] I [client.c:2103:client_rpc_notify] 1-dr-store-client-0: disconnected from 10.70.37.86:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:56.634337] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-5: server 10.70.37.187:49155 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 21:35:56.638141] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-5: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:02.322954 (xid=0x2670x)
>[2014-02-04 21:35:56.638164] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:35:56.638239] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-5: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:06.473527 (xid=0x2671x)
>[2014-02-04 21:35:56.638250] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:35:56.638289] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-5: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-02-04 21:35:14.545112 (xid=0x2672x)
>[2014-02-04 21:35:56.638297] W [client-handshake.c:276:client_ping_cbk] 1-dr-store-client-5: timer must have expired
>[2014-02-04 21:35:56.638316] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-5: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:18.595628 (xid=0x2673x)
>[2014-02-04 21:35:56.638325] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-5: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:35:56.638359] I [client.c:2103:client_rpc_notify] 1-dr-store-client-5: disconnected from 10.70.37.187:49155. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:35:56.638371] E [afr-common.c:4025:afr_notify] 1-dr-store-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:36:07.303825] E [socket.c:2158:socket_connect_finish] 1-dr-store-client-5: connection to 10.70.37.187:24007 failed (No route to host)
>[2014-02-04 21:36:08.677636] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-3: server 10.70.37.187:49153 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 21:36:08.678370] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS 3.3) op(FSYNC(16)) called at 2014-02-04 21:35:00.542502 (xid=0x118726x)
>[2014-02-04 21:36:08.678385] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:08.678409] W [afr-transaction.c:1553:afr_changelog_fsync_cbk] 1-dr-store-replicate-1: fsync(b31d8277-fae7-48c7-a377-1b8fc817a6eb) failed on subvolume dr-store-client-3. Transaction was WRITE
>[2014-02-04 21:36:08.678442] W [client-rpc-fops.c:1811:client3_3_fxattrop_cbk] 1-dr-store-client-2: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:08.681225] I [socket.c:3032:socket_submit_request] 1-dr-store-client-3: not connected (priv->connected = 0)
>[2014-02-04 21:36:08.681242] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-3: failed to submit rpc-request (XID: 0x118732x Program: GlusterFS 3.3, ProgVers: 330, Proc: 34) to rpc-transport (dr-store-client-3)
>[2014-02-04 21:36:08.681254] W [client-rpc-fops.c:1811:client3_3_fxattrop_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:08.681297] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:02.322859 (xid=0x118727x)
>[2014-02-04 21:36:08.681309] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:08.681372] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2014-02-04 21:35:02.979839 (xid=0x118728x)
>[2014-02-04 21:36:08.681381] W [client-rpc-fops.c:1811:client3_3_fxattrop_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:08.684103] I [socket.c:3032:socket_submit_request] 1-dr-store-client-2: not connected (priv->connected = 0)
>[2014-02-04 21:36:08.684120] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-2: failed to submit rpc-request (XID: 0x118408x Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (dr-store-client-2)
>[2014-02-04 21:36:08.684140] W [client-rpc-fops.c:866:client3_3_writev_cbk] 1-dr-store-client-2: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:08.684169] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070038: WRITE => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:08.684206] W [client-rpc-fops.c:4034:client3_3_writev] 1-dr-store-client-2: failed to send the fop
>[2014-02-04 21:36:08.684236] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:06.473471 (xid=0x118729x)
>[2014-02-04 21:36:08.684247] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:08.684300] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:18.595568 (xid=0x118730x)
>[2014-02-04 21:36:08.684310] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:08.684356] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-3: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-02-04 21:35:26.547195 (xid=0x118731x)
>[2014-02-04 21:36:08.684363] W [client-handshake.c:276:client_ping_cbk] 1-dr-store-client-3: timer must have expired
>[2014-02-04 21:36:08.684374] I [client.c:2103:client_rpc_notify] 1-dr-store-client-3: disconnected from 10.70.37.187:49153. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:36:08.684383] E [afr-common.c:4025:afr_notify] 1-dr-store-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:36:10.305842] E [socket.c:2158:socket_connect_finish] 1-dr-store-client-3: connection to 10.70.37.187:24007 failed (No route to host)
>[2014-02-04 21:36:10.682405] W [client-rpc-fops.c:1811:client3_3_fxattrop_cbk] 1-dr-store-client-2: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:10.682439] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-2: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:10.682461] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-3: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.694423] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-1: server 10.70.37.187:49152 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 21:36:12.696771] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2014-02-04 21:35:01.324262 (xid=0x232638x)
>[2014-02-04 21:36:12.696793] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.696837] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.696866] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702215] I [socket.c:3032:socket_submit_request] 1-dr-store-client-1: not connected (priv->connected = 0)
>[2014-02-04 21:36:12.702233] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-1: failed to submit rpc-request (XID: 0x232646x Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (dr-store-client-1)
>[2014-02-04 21:36:12.702246] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702255] I [afr-lk-common.c:1092:afr_lock_blocking] 1-dr-store-replicate-0: unable to lock on even one child
>[2014-02-04 21:36:12.702263] I [afr-transaction.c:1119:afr_post_blocking_inodelk_cbk] 1-dr-store-replicate-0: Blocking inodelks failed.
>[2014-02-04 21:36:12.702294] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070033: WRITE => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:12.702358] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2014-02-04 21:35:02.859215 (xid=0x232640x)
>[2014-02-04 21:36:12.702367] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702382] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702392] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702409] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-1: failed to submit rpc-request (XID: 0x232647x Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (dr-store-client-1)
>[2014-02-04 21:36:12.702419] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702427] I [afr-lk-common.c:1092:afr_lock_blocking] 1-dr-store-replicate-0: unable to lock on even one child
>[2014-02-04 21:36:12.702434] I [afr-transaction.c:1119:afr_post_blocking_inodelk_cbk] 1-dr-store-replicate-0: Blocking inodelks failed.
>[2014-02-04 21:36:12.702459] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070036: WRITE => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:12.702509] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2014-02-04 21:35:03.565655 (xid=0x232641x)
>[2014-02-04 21:36:12.702517] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702531] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702541] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702570] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-1: failed to submit rpc-request (XID: 0x232648x Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (dr-store-client-1)
>[2014-02-04 21:36:12.702580] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 21:36:12.702595] I [afr-lk-common.c:1092:afr_lock_blocking] 1-dr-store-replicate-0: unable to lock on even one child
>[2014-02-04 21:36:12.702603] I [afr-transaction.c:1119:afr_post_blocking_inodelk_cbk] 1-dr-store-replicate-0: Blocking inodelks failed.
>[2014-02-04 21:36:12.702624] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070040: WRITE => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:12.702669] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:02.322793 (xid=0x232639x)
>[2014-02-04 21:36:12.702681] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:12.702834] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:06.473420 (xid=0x232642x)
>[2014-02-04 21:36:12.702846] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:12.702927] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:18.595508 (xid=0x232643x)
>[2014-02-04 21:36:12.702949] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:12.703013] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2014-02-04 21:35:30.547982 (xid=0x232644x)
>[2014-02-04 21:36:12.703021] W [client-handshake.c:276:client_ping_cbk] 1-dr-store-client-1: timer must have expired
>[2014-02-04 21:36:12.703042] E [rpc-clnt.c:369:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x7fa34cfca524] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fa34cfca063] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fa34cfc9f7e]))) 1-dr-store-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2014-02-04 21:35:33.605277 (xid=0x232645x)
>[2014-02-04 21:36:12.703051] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:12.703101] W [rpc-clnt.c:1503:rpc_clnt_submit] 1-dr-store-client-1: failed to submit rpc-request (XID: 0x232649x Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (dr-store-client-1)
>[2014-02-04 21:36:12.703115] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001)
>[2014-02-04 21:36:12.703152] W [fuse-bridge.c:1134:fuse_attr_cbk] 0-glusterfs-fuse: 17070043: LOOKUP() / => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:12.703204] I [client.c:2103:client_rpc_notify] 1-dr-store-client-1: disconnected from 10.70.37.187:49152. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:36:12.703216] E [afr-common.c:4025:afr_notify] 1-dr-store-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-02-04 21:36:12.704123] I [afr-common.c:4150:afr_local_init] 1-dr-store-replicate-0: no subvolumes up
>[2014-02-04 21:36:12.711201] W [fuse-bridge.c:568:fuse_entry_cbk] 0-glusterfs-fuse: 17070062: LOOKUP() /20d80bc2-4664-46fb-88df-ae82a04cf535 => -1 (Transport endpoint is not connected)
>[2014-02-04 21:36:12.711410] W [defaults.c:1291:default_release] (-->/usr/lib64/glusterfs/3.4.0.58rhs/xlator/mount/fuse.so(+0x22d48) [0x7fa34b25bd48] (-->/usr/lib64/glusterfs/3.4.0.58rhs/xlator/mount/fuse.so(+0xbc0a) [0x7fa34b244c0a] (-->/usr/lib64/libglusterfs.so.0(fd_unref+0x144) [0x7fa34d20e6a4]))) 0-fuse: xlator does not implement release_cbk
>[2014-02-04 21:36:13.351930] E [socket.c:2158:socket_connect_finish] 1-dr-store-client-1: connection to 10.70.37.187:24007 failed (No route to host)
>[2014-02-04 21:45:21.075192] W [fuse-bridge.c:1134:fuse_attr_cbk] 0-glusterfs-fuse: 17070106: LOOKUP() / => -1 (Transport endpoint is not connected)
>[2014-02-04 21:54:45.162839] W [fuse-bridge.c:1134:fuse_attr_cbk] 0-glusterfs-fuse: 17070148: LOOKUP() / => -1 (Transport endpoint is not connected)
>[2014-02-04 21:56:28.504098] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:56:28.504175] I [client.c:2103:client_rpc_notify] 1-dr-store-client-5: disconnected from 10.70.37.187:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:56:28.505171] E [client-handshake.c:1778:client_query_portmap_cbk] 1-dr-store-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-04 21:56:28.505221] I [client.c:2103:client_rpc_notify] 1-dr-store-client-1: disconnected from 10.70.37.187:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-04 21:56:30.846739] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-5: changing port to 49155 (from 0)
>[2014-02-04 21:56:30.850257] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:30.850681] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:30.850702] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-5: Connected to 10.70.37.187:49155, attached to remote volume '/rhs/brick3/dsr3'.
>[2014-02-04 21:56:30.850711] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:30.850780] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-2: Subvolume 'dr-store-client-5' came back up; going online.
>[2014-02-04 21:56:30.851018] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-5: Server lk version = 1
>[2014-02-04 21:56:31.849443] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-1: changing port to 49152 (from 0)
>[2014-02-04 21:56:31.854294] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:31.854733] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:31.854751] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-1: Connected to 10.70.37.187:49152, attached to remote volume '/rhs/brick1/dsr1'.
>[2014-02-04 21:56:31.854760] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:31.854790] I [client-handshake.c:1314:client_post_handshake] 1-dr-store-client-1: 3 fds open - Delaying child_up until they are re-opened
>[2014-02-04 21:56:31.867722] I [client-handshake.c:936:client_child_up_reopen_done] 1-dr-store-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2014-02-04 21:56:31.867757] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-0: Subvolume 'dr-store-client-1' came back up; going online.
>[2014-02-04 21:56:31.868060] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-1: Server lk version = 1
>[2014-02-04 21:56:33.863874] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-3: changing port to 49153 (from 0)
>[2014-02-04 21:56:33.867449] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:33.868098] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:33.868120] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-3: Connected to 10.70.37.187:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-04 21:56:33.868129] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:33.868136] I [client-handshake.c:1314:client_post_handshake] 1-dr-store-client-3: 1 fds open - Delaying child_up until they are re-opened
>[2014-02-04 21:56:33.868790] I [client-handshake.c:936:client_child_up_reopen_done] 1-dr-store-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2014-02-04 21:56:33.868832] I [afr-common.c:3988:afr_notify] 1-dr-store-replicate-1: Subvolume 'dr-store-client-3' came back up; going online.
>[2014-02-04 21:56:33.869168] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-3: Server lk version = 1
>[2014-02-04 21:56:39.522835] W [afr-inode-write.c:510:afr_writev] 1-dr-store-replicate-0: failing writev due to lack of quorum
>[2014-02-04 21:56:39.522869] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070203: WRITE => -1 (Read-only file system)
>[2014-02-04 21:56:39.536886] W [afr-inode-write.c:510:afr_writev] 1-dr-store-replicate-1: failing writev due to lack of quorum
>[2014-02-04 21:56:39.536913] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070207: WRITE => -1 (Read-only file system)
>[2014-02-04 21:56:39.726681] W [afr-inode-write.c:510:afr_writev] 1-dr-store-replicate-0: failing writev due to lack of quorum
>[2014-02-04 21:56:39.726711] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070250: WRITE => -1 (Read-only file system)
>[2014-02-04 21:56:39.838559] W [afr-inode-write.c:510:afr_writev] 1-dr-store-replicate-0: failing writev due to lack of quorum
>[2014-02-04 21:56:39.838589] W [fuse-bridge.c:2696:fuse_writev_cbk] 0-glusterfs-fuse: 17070252: WRITE => -1 (Read-only file system)
>[2014-02-04 21:56:46.903560] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-2: changing port to 49153 (from 0)
>[2014-02-04 21:56:46.908532] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-4: changing port to 49155 (from 0)
>[2014-02-04 21:56:46.913436] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:46.913936] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:46.913962] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-2: Connected to 10.70.37.86:49153, attached to remote volume '/rhs/brick2/dsr2'.
>[2014-02-04 21:56:46.913973] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:46.913980] I [client-handshake.c:1314:client_post_handshake] 1-dr-store-client-2: 1 fds open - Delaying child_up until they are re-opened
>[2014-02-04 21:56:46.914085] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:46.914446] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 1-dr-store-client-0: changing port to 49152 (from 0)
>[2014-02-04 21:56:46.917405] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:46.917423] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-4: Connected to 10.70.37.86:49155, attached to remote volume '/rhs/brick3/dsr3'.
>[2014-02-04 21:56:46.917431] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:46.917501] I [client-handshake.c:936:client_child_up_reopen_done] 1-dr-store-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2014-02-04 21:56:46.917793] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-2: Server lk version = 1
>[2014-02-04 21:56:46.917941] I [client-handshake.c:1695:select_server_supported_programs] 1-dr-store-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-04 21:56:46.918317] I [client-handshake.c:1466:client_setvolume_cbk] 1-dr-store-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-04 21:56:46.918338] I [client-handshake.c:1480:client_setvolume_cbk] 1-dr-store-client-0: Connected to 10.70.37.86:49152, attached to remote volume '/rhs/brick1/drs1'.
>[2014-02-04 21:56:46.918347] I [client-handshake.c:1492:client_setvolume_cbk] 1-dr-store-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-04 21:56:46.918355] I [client-handshake.c:1314:client_post_handshake] 1-dr-store-client-0: 3 fds open - Delaying child_up until they are re-opened
>[2014-02-04 21:56:46.919277] I [client-handshake.c:936:client_child_up_reopen_done] 1-dr-store-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2014-02-04 21:56:46.919327] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-4: Server lk version = 1
>[2014-02-04 21:56:46.919614] I [client-handshake.c:450:client_set_lk_version_cbk] 1-dr-store-client-0: Server lk version = 1
>[2014-02-04 22:17:24.095650] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-1: server 10.70.37.187:49152 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 22:17:47.099842] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-2: server 10.70.37.86:49153 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 22:17:48.099986] C [client-handshake.c:127:rpc_client_ping_timer_expired] 1-dr-store-client-0: server 10.70.37.86:49152 has not responded in the last 42 seconds, disconnecting.
>[2014-02-04 22:46:42.227994] E [rpc-clnt.c:208:call_bail] 1-dr-store-client-1: bailing out frame type(GlusterFS 3.3) op(FSYNC(16)) xid = 0x234866x sent = 2014-02-04 22:16:38.497154. timeout = 1800 for 10.70.37.187:49152
>[2014-02-04 22:46:42.228049] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 1-dr-store-client-1: remote operation failed: Transport endpoint is not connected
>[2014-02-04 22:46:42.228069] W [afr-transaction.c:1553:afr_changelog_fsync_cbk] 1-dr-store-replicate-0: fsync(2287b21f-dc5e-47f5-8730-679ee9b44273) failed on subvolume dr-store-client-1. Transaction was WRITE
>[2014-02-04 22:46:48.228531] E [rpc-clnt.c:208:call_bail] 1-dr-store-client-0: bailing out frame type(GlusterFS 3.3) op(FSYNC(16)) xid = 0x235871x sent = 2014-02-04 22:16:38.497142. timeout = 1800 for 10.70.37.86:49152
>[2014-02-04 22:46:48.228592] W [client-rpc-fops.c:984:client3_3_fsync_cbk] 1-dr-store-client-0: remote operation failed: Transport endpoint is not connected
>[2014-02-04 22:46:48.228612] W [afr-transaction.c:1553:afr_changelog_fsync_cbk] 1-dr-store-replicate-0: fsync(2287b21f-dc5e-47f5-8730-679ee9b44273) failed on subvolume dr-store-client-0. Transaction was WRITE

Actions: View

Attachments on bug 1061580: 859496 | 859498 | 859499 | 859501 | 859502 | 860207