Attachment 1128470 Details for Bug 1309984 – ganesha-gfapi.log

ganesha-gfapi.log

ganesha-gfapi.log (text/plain), 54.98 KB, created by Shashank Raj on 2016-02-19 07:05:26 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Shashank Raj

Created: 2016-02-19 07:05:26 UTC

Size: 54.98 KB

patch

obsolete

>[2016-02-18 01:03:16.612322] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
>[2016-02-18 01:03:16.671843] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
>[2016-02-18 01:03:16.676627] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e667331-2d33-3534-312d-323031362d30 (0) coming up
>[2016-02-18 01:03:16.676700] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-0: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.680116] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-1: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.680477] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-0: changing port to 49153 (from 0)
>[2016-02-18 01:03:16.683277] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-2: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.688752] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.689088] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-0: Connected to testvolume-client-0, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 01:03:16.689110] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.689171] I [MSGID: 108005] [afr-common.c:3960:afr_notify] 0-testvolume-replicate-0: Subvolume 'testvolume-client-0' came back up; going online.
>[2016-02-18 01:03:16.689304] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-0: Server lk version = 1
>[2016-02-18 01:03:16.691144] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-3: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.694345] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-4: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.697661] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-5: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.698350] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-4: changing port to 49154 (from 0)
>[2016-02-18 01:03:16.700924] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-6: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.706618] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.706885] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-4: Connected to testvolume-client-4, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 01:03:16.706925] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.706991] I [MSGID: 108005] [afr-common.c:3960:afr_notify] 0-testvolume-replicate-1: Subvolume 'testvolume-client-4' came back up; going online.
>[2016-02-18 01:03:16.707025] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-4: Server lk version = 1
>[2016-02-18 01:03:16.709319] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-7: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.712545] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-8: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.715876] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-9: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.716261] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-8: changing port to 49155 (from 0)
>[2016-02-18 01:03:16.719184] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-10: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.724589] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.725136] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-8: Connected to testvolume-client-8, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 01:03:16.725177] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.725323] I [MSGID: 108005] [afr-common.c:3960:afr_notify] 0-testvolume-replicate-2: Subvolume 'testvolume-client-8' came back up; going online.
>[2016-02-18 01:03:16.725500] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-8: Server lk version = 1
>[2016-02-18 01:03:16.727363] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-11: parent translators are ready, attempting connect on transport
>[2016-02-18 01:03:16.748865] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-2: this function should not be called
>[2016-02-18 01:03:16.774209] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-1: this function should not be called
>[2016-02-18 01:03:16.777607] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-3: this function should not be called
>[2016-02-18 01:03:16.828393] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-2: this function should not be called
>[2016-02-18 01:03:16.831468] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-2: changing port to 49153 (from 0)
>[2016-02-18 01:03:16.835203] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-6: changing port to 49154 (from 0)
>[2016-02-18 01:03:16.837947] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-10: changing port to 49155 (from 0)
>The message "W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-2: this function should not be called" repeated 2 times between [2016-02-18 01:03:16.828393] and [2016-02-18 01:03:16.829964]
>[2016-02-18 01:03:16.841261] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.841512] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.841711] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-2: Connected to testvolume-client-2, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 01:03:16.841731] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.841899] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-6: Connected to testvolume-client-6, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 01:03:16.841919] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.841986] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-2: Server lk version = 1
>[2016-02-18 01:03:16.842120] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-6: Server lk version = 1
>[2016-02-18 01:03:16.845016] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.845371] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-10: Connected to testvolume-client-10, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 01:03:16.845400] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.845459] I [MSGID: 108005] [afr-common.c:3960:afr_notify] 0-testvolume-replicate-3: Subvolume 'testvolume-client-10' came back up; going online.
>[2016-02-18 01:03:16.845586] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-10: Server lk version = 1
>[2016-02-18 01:03:16.849481] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-3: this function should not be called
>The message "W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-3: this function should not be called" repeated 2 times between [2016-02-18 01:03:16.849481] and [2016-02-18 01:03:16.850237]
>[2016-02-18 01:03:16.851766] W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-1: this function should not be called
>[2016-02-18 01:03:16.851816] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-3: changing port to 49153 (from 0)
>[2016-02-18 01:03:16.854611] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-1: changing port to 49152 (from 0)
>[2016-02-18 01:03:16.856654] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-7: changing port to 49154 (from 0)
>The message "W [MSGID: 114010] [client-callback.c:33:client_cbk_fetchspec] 0-testvolume-client-1: this function should not be called" repeated 2 times between [2016-02-18 01:03:16.851766] and [2016-02-18 01:03:16.852432]
>[2016-02-18 01:03:16.861452] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.861569] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-11: changing port to 49155 (from 0)
>[2016-02-18 01:03:16.863683] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-3: Connected to testvolume-client-3, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 01:03:16.863709] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.863794] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.863868] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-5: changing port to 49153 (from 0)
>[2016-02-18 01:03:16.863918] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-testvolume-client-9: changing port to 49154 (from 0)
>[2016-02-18 01:03:16.864005] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-3: Server lk version = 1
>[2016-02-18 01:03:16.870437] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-1: Connected to testvolume-client-1, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 01:03:16.870463] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.870658] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.870801] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-1: Server lk version = 1
>[2016-02-18 01:03:16.871029] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.871116] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.871213] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-7: Connected to testvolume-client-7, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 01:03:16.871248] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.871445] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-7: Server lk version = 1
>[2016-02-18 01:03:16.871490] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-11: Connected to testvolume-client-11, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 01:03:16.871506] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.871605] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-5: Connected to testvolume-client-5, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 01:03:16.871622] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.871808] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-11: Server lk version = 1
>[2016-02-18 01:03:16.871853] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-5: Server lk version = 1
>[2016-02-18 01:03:16.874486] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 01:03:16.874869] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-9: Connected to testvolume-client-9, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 01:03:16.874891] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 01:03:16.885097] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-9: Server lk version = 1
>[2016-02-18 01:03:16.887798] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-0: selecting local read_child testvolume-client-0
>[2016-02-18 01:03:16.889269] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-2: selecting local read_child testvolume-client-8
>[2016-02-18 01:03:16.889629] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-1: selecting local read_child testvolume-client-4
>[2016-02-18 01:03:16.889985] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-testvolume-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
>[2016-02-18 01:03:16.900727] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-2: selecting local read_child testvolume-client-8
>[2016-02-18 01:03:16.901135] I [MSGID: 109036] [dht-common.c:7106:dht_log_new_layout_for_dir_selfheal] 0-testvolume-dht: Setting layout of / with [Subvol_name: testvolume-replicate-0, Err: -1 , Start: 0 , Stop: 1073727049 , Hash: 1 ], [Subvol_name: testvolume-replicate-1, Err: -1 , Start: 1073727050 , Stop: 2147454099 , Hash: 1 ], [Subvol_name: testvolume-replicate-2, Err: -1 , Start: 2147454100 , Stop: 3221181149 , Hash: 1 ], [Subvol_name: testvolume-replicate-3, Err: -1 , Start: 3221181150 , Stop: 4294967295 , Hash: 1 ], 
>[2016-02-18 01:03:16.901268] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-0: selecting local read_child testvolume-client-0
>[2016-02-18 01:03:16.901844] I [MSGID: 108031] [afr-common.c:1745:afr_local_discovery_cbk] 0-testvolume-replicate-1: selecting local read_child testvolume-client-4
>[2016-02-18 01:03:16.903458] I [MSGID: 104041] [glfs-resolve.c:843:__glfs_active_subvol] 0-testvolume: switched to graph 6e667331-2d33-3534-312d-323031362d30 (0)
>[2016-02-18 02:12:32.845153] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
>[2016-02-18 02:12:32.898192] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
>[2016-02-18 02:12:32.903165] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6e667331-2d32-3839-3530-2d323031362d (0) coming up
>[2016-02-18 02:12:32.903230] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-0: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.906584] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-1: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.907136] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-0: changing port to 49153 (from 0)
>[2016-02-18 02:12:32.909732] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-2: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.916600] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.916776] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-1: changing port to 49152 (from 0)
>[2016-02-18 02:12:32.919365] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-3: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.925387] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-0: Connected to testvolume-client-0, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 02:12:32.925418] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.925550] I [MSGID: 108005] [afr-common.c:3976:afr_notify] 0-testvolume-replicate-0: Subvolume 'testvolume-client-0' came back up; going online.
>[2016-02-18 02:12:32.926044] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-0: Server lk version = 1
>[2016-02-18 02:12:32.926116] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.926457] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-2: changing port to 49153 (from 0)
>[2016-02-18 02:12:32.926603] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-1: Connected to testvolume-client-1, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 02:12:32.926627] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.929245] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-4: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.935450] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-1: Server lk version = 1
>[2016-02-18 02:12:32.936068] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.936474] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-3: changing port to 49153 (from 0)
>[2016-02-18 02:12:32.936601] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-2: Connected to testvolume-client-2, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 02:12:32.936637] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.940797] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-5: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.946587] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-2: Server lk version = 1
>[2016-02-18 02:12:32.947050] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.947124] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-4: changing port to 49154 (from 0)
>[2016-02-18 02:12:32.949959] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-6: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.953118] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-3: Connected to testvolume-client-3, attached to remote volume '/bricks/brick0/b0'.
>[2016-02-18 02:12:32.953145] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.953206] I [MSGID: 108005] [afr-common.c:3976:afr_notify] 0-testvolume-replicate-1: Subvolume 'testvolume-client-3' came back up; going online.
>[2016-02-18 02:12:32.953411] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-3: Server lk version = 1
>[2016-02-18 02:12:32.953619] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-5: changing port to 49153 (from 0)
>[2016-02-18 02:12:32.953668] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.956124] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-7: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.960700] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-4: Connected to testvolume-client-4, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 02:12:32.960738] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.961057] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-4: Server lk version = 1
>[2016-02-18 02:12:32.961672] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.962310] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-5: Connected to testvolume-client-5, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 02:12:32.962351] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.962842] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-6: changing port to 49154 (from 0)
>[2016-02-18 02:12:32.963018] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-5: Server lk version = 1
>[2016-02-18 02:12:32.965515] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-8: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.971061] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.971303] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-7: changing port to 49154 (from 0)
>[2016-02-18 02:12:32.976160] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-9: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.981558] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-6: Connected to testvolume-client-6, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 02:12:32.981668] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.981824] I [MSGID: 108005] [afr-common.c:3976:afr_notify] 0-testvolume-replicate-2: Subvolume 'testvolume-client-6' came back up; going online.
>[2016-02-18 02:12:32.982174] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-6: Server lk version = 1
>[2016-02-18 02:12:32.982518] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.982826] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-8: changing port to 49155 (from 0)
>[2016-02-18 02:12:32.986857] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-10: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.993787] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-7: Connected to testvolume-client-7, attached to remote volume '/bricks/brick1/b1'.
>[2016-02-18 02:12:32.993834] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.994291] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-7: Server lk version = 1
>[2016-02-18 02:12:32.994708] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-9: changing port to 49154 (from 0)
>[2016-02-18 02:12:32.994814] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:32.995701] I [MSGID: 114020] [client.c:2118:notify] 0-testvolume-client-11: parent translators are ready, attempting connect on transport
>[2016-02-18 02:12:32.999309] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-8: Connected to testvolume-client-8, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 02:12:32.999372] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:32.999807] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-8: Server lk version = 1
>[2016-02-18 02:12:32.999899] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:33.000358] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-10: changing port to 49155 (from 0)
>[2016-02-18 02:12:33.000437] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-9: Connected to testvolume-client-9, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 02:12:33.000455] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:33.000523] I [MSGID: 108005] [afr-common.c:3976:afr_notify] 0-testvolume-replicate-3: Subvolume 'testvolume-client-9' came back up; going online.
>[2016-02-18 02:12:33.004905] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-9: Server lk version = 1
>[2016-02-18 02:12:33.005452] I [rpc-clnt.c:1851:rpc_clnt_reconfig] 0-testvolume-client-11: changing port to 49155 (from 0)
>[2016-02-18 02:12:33.010099] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:33.010774] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-10: Connected to testvolume-client-10, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 02:12:33.010805] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:33.011137] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-10: Server lk version = 1
>[2016-02-18 02:12:33.014940] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-testvolume-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2016-02-18 02:12:33.015626] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-testvolume-client-11: Connected to testvolume-client-11, attached to remote volume '/bricks/brick2/b2'.
>[2016-02-18 02:12:33.015656] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-testvolume-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2016-02-18 02:12:33.029261] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-testvolume-client-11: Server lk version = 1
>[2016-02-18 02:12:33.031556] I [MSGID: 108031] [afr-common.c:1883:afr_local_discovery_cbk] 0-testvolume-replicate-0: selecting local read_child testvolume-client-0
>[2016-02-18 02:12:33.034130] I [MSGID: 108031] [afr-common.c:1883:afr_local_discovery_cbk] 0-testvolume-replicate-1: selecting local read_child testvolume-client-4
>[2016-02-18 02:12:33.035143] I [MSGID: 104041] [glfs-resolve.c:869:__glfs_active_subvol] 0-testvolume: switched to graph 6e667331-2d32-3839-3530-2d323031362d (0)
>[2016-02-18 02:12:33.035303] I [MSGID: 108031] [afr-common.c:1883:afr_local_discovery_cbk] 0-testvolume-replicate-2: selecting local read_child testvolume-client-8
>[2016-02-18 02:22:18.486737] W [socket.c:588:__socket_rwv] 0-testvolume-client-1: readv on 10.70.37.82:49152 failed (No data available)
>[2016-02-18 02:22:18.486792] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-1: disconnected from testvolume-client-1. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:22:18.489344] W [socket.c:588:__socket_rwv] 0-testvolume-client-5: readv on 10.70.37.82:49153 failed (No data available)
>[2016-02-18 02:22:18.489389] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-5: disconnected from testvolume-client-5. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:22:18.490576] W [socket.c:588:__socket_rwv] 0-testvolume-client-9: readv on 10.70.37.82:49154 failed (No data available)
>[2016-02-18 02:22:18.490611] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-9: disconnected from testvolume-client-9. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:22:28.951814] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-1: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:22:28.954818] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-5: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:22:28.957953] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-9: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:27:20.330050] W [socket.c:588:__socket_rwv] 0-testvolume-client-7: readv on 10.70.37.46:49154 failed (Connection timed out)
>[2016-02-18 02:27:20.330050] W [socket.c:588:__socket_rwv] 0-testvolume-client-11: readv on 10.70.37.46:49155 failed (Connection timed out)
>[2016-02-18 02:27:20.330171] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-7: disconnected from testvolume-client-7. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:20.330181] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-11: disconnected from testvolume-client-11. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:20.330254] W [MSGID: 108001] [afr-common.c:4059:afr_notify] 0-testvolume-replicate-3: Client-quorum is not met
>[2016-02-18 02:27:20.757727] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvolume-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2016-02-18 02:27:20.758074] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-9: disconnected from testvolume-client-9. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:20.758809] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-testvolume-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2016-02-18 02:27:20.759078] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-1: disconnected from testvolume-client-1. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:20.759506] W [MSGID: 114032] [client-handshake.c:1506:client_query_portmap_cbk] 0-testvolume-client-5: received RPC status error, try again later [Transport endpoint is not connected]
>[2016-02-18 02:27:20.759651] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-5: disconnected from testvolume-client-5. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:23.549534] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-9: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:27:23.552211] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-1: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:27:23.555348] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-5: connection to 10.70.37.82:24007 failed (Connection refused)
>[2016-02-18 02:27:50.675369] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-testvolume-client-3: server 10.70.37.46:49153 has not responded in the last 42 seconds, disconnecting.
>[2016-02-18 02:27:50.688529] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2016-02-18 02:26:38.999042 (xid=0x896)
>[2016-02-18 02:27:50.688560] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.689139] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2016-02-18 02:26:48.864440 (xid=0x8a0)
>[2016-02-18 02:27:50.689557] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2016-02-18 02:26:50.429511 (xid=0x8a1)
>[2016-02-18 02:27:50.689742] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:38.998307 (xid=0x89a)
>The message "E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2016-02-18 02:27:50.688560] and [2016-02-18 02:27:50.689587]
>[2016-02-18 02:27:50.689770] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.693809] I [socket.c:3308:socket_submit_request] 0-testvolume-client-3: not connected (priv->connected = 0)
>[2016-02-18 02:27:50.693840] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.693853] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.693876] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694163] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:38.998991 (xid=0x89b)
>[2016-02-18 02:27:50.694187] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694244] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694259] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694277] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694523] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:38.999003 (xid=0x895)
>[2016-02-18 02:27:50.694545] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694551] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.694597] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694613] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694625] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694633] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694684] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694828] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.694861] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694872] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694898] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:38.999860 (xid=0x89c)
>[2016-02-18 02:27:50.694921] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694962] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694975] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.694989] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.695029] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.694992] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.695049] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.695198] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.695213] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.695229] W [MSGID: 114029] [client-rpc-fops.c:4415:client3_3_writev] 0-testvolume-client-3: failed to send the fop
>[2016-02-18 02:27:50.695263] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:39.000628 (xid=0x89d)
>[2016-02-18 02:27:50.695291] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.695340] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.695354] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.695371] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696263] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.696395] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:39.001241 (xid=0x89e)
>[2016-02-18 02:27:50.696427] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696466] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.696479] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696496] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696750] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:39.001540 (xid=0x89f)
>[2016-02-18 02:27:50.696771] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696815] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8af Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.696828] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.696845] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697060] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:50.429695 (xid=0x8a2)
>[2016-02-18 02:27:50.697087] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697130] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.697142] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697159] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697635] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2016-02-18 02:26:50.430710 (xid=0x8a3)
>[2016-02-18 02:27:50.697657] W [MSGID: 114031] [client-rpc-fops.c:907:client3_3_writev_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697702] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.697716] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697724] W [rpc-clnt.c:1588:rpc_clnt_submit] 0-testvolume-client-3: failed to submit rpc-request (XID: 0x8b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 16) to rpc-transport (testvolume-client-3)
>[2016-02-18 02:27:50.697755] W [MSGID: 114031] [client-rpc-fops.c:1031:client3_3_fsync_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697760] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697960] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1eb)[0x7fb74a4a16eb] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7fb74a26c227] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb74a26c33e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7fb74a26c40b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1c2)[0x7fb74a26ca42] ))))) 0-testvolume-client-3: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2016-02-18 02:27:08.493240 (xid=0x8a4)
>[2016-02-18 02:27:50.697982] W [rpc-clnt-ping.c:208:rpc_clnt_ping_cbk] 0-testvolume-client-3: socket disconnected
>[2016-02-18 02:27:50.697768] W [MSGID: 108035] [afr-transaction.c:1611:afr_changelog_fsync_cbk] 0-testvolume-replicate-1: fsync(92989e49-afca-4555-89f9-8718dc8880fb) failed on subvolume testvolume-client-3. Transaction was WRITE [Transport endpoint is not connected]
>[2016-02-18 02:27:50.697998] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-testvolume-client-3: disconnected from testvolume-client-3. Client process will keep trying to connect to glusterd until brick's port is available
>[2016-02-18 02:27:50.698050] W [MSGID: 108001] [afr-common.c:4059:afr_notify] 0-testvolume-replicate-1: Client-quorum is not met
>[2016-02-18 02:27:50.699668] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.699696] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.699786] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.699815] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.700003] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.700045] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.700170] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.700191] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.700356] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.700379] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:50.700454] W [MSGID: 108001] [afr-transaction.c:686:afr_handle_quorum] 0-testvolume-replicate-1: 92989e49-afca-4555-89f9-8718dc8880fb: Failing WRITE as quorum is not met
>[2016-02-18 02:27:50.700475] E [MSGID: 114031] [client-rpc-fops.c:1676:client3_3_finodelk_cbk] 0-testvolume-client-3: remote operation failed [Transport endpoint is not connected]
>[2016-02-18 02:27:51.588126] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-3: connection to 10.70.37.46:24007 failed (No route to host)
>[2016-02-18 02:28:33.586053] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-7: connection to 10.70.37.46:24007 failed (Connection timed out)
>[2016-02-18 02:28:33.589269] E [socket.c:2278:socket_connect_finish] 0-testvolume-client-11: connection to 10.70.37.46:24007 failed (Connection timed out)

Actions: View

Attachments on bug 1309984: 1128469 | 1128470