Attachment 852843 Details for Bug 1055747 – glusterd log file from one of the nodes showing the locking issue.

glusterd log file from one of the nodes showing the locking issue.

etc-glusterfs-glusterd.vol.log (text/plain), 44.04 KB, created by Paul Cuzner on 2014-01-20 20:47:45 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Paul Cuzner

Created: 2014-01-20 20:47:45 UTC

Size: 44.04 KB

patch

obsolete

>[2014-01-20 05:55:14.782801] I [glusterfsd.c:1942:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0beta1 (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-01-20 05:55:15.025489] I [glusterd.c:1131:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-01-20 05:55:15.026457] I [rpcsvc.c:2060:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-01-20 05:55:15.028647] I [socket.c:3533:socket_init] 0-socket.management: SSL support is NOT enabled
>[2014-01-20 05:55:15.028671] I [socket.c:3548:socket_init] 0-socket.management: using system polling thread
>[2014-01-20 05:55:15.032874] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.0beta1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
>[2014-01-20 05:55:15.032896] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
>[2014-01-20 05:55:15.032903] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-01-20 05:55:15.032919] I [rpcsvc.c:2060:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-01-20 05:55:15.032982] I [socket.c:3533:socket_init] 0-socket.management: SSL support is NOT enabled
>[2014-01-20 05:55:15.032990] I [socket.c:3548:socket_init] 0-socket.management: using system polling thread
>[2014-01-20 05:55:15.045494] I [glusterd.c:367:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
>[2014-01-20 05:55:15.046380] E [store.c:408:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-01-20 05:55:15.046408] E [store.c:408:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-01-20 05:55:15.046417] I [glusterd-store.c:1441:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 3
>[2014-01-20 05:55:15.046575] E [store.c:408:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory)
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13: 
>+------------------------------------------------------------------------------+
>[2014-01-20 05:56:45.824711] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-2 24007
>[2014-01-20 05:56:45.843140] I [glusterd-handler.c:2931:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: glfs35-2 (24007)
>[2014-01-20 05:56:46.473437] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-01-20 05:56:46.473556] I [socket.c:3533:socket_init] 0-management: SSL support is NOT enabled
>[2014-01-20 05:56:46.473567] I [socket.c:3548:socket_init] 0-management: using system polling thread
>[2014-01-20 05:56:46.478829] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
>[2014-01-20 05:56:47.262671] E [store.c:408:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-01-20 05:56:47.262862] I [glusterd.c:168:glusterd_uuid_generate_save] 0-management: generated UUID: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 05:56:48.645956] I [glusterd-rpc-ops.c:234:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 05:56:49.640329] I [glusterd-rpc-ops.c:306:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-01-20 05:56:49.640505] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2, port: 0
>[2014-01-20 05:56:50.553007] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-3 24007
>[2014-01-20 05:56:50.555400] I [glusterd-handler.c:2931:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: glfs35-3 (24007)
>[2014-01-20 05:56:51.173727] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-01-20 05:56:51.173812] I [socket.c:3533:socket_init] 0-management: SSL support is NOT enabled
>[2014-01-20 05:56:51.173823] I [socket.c:3548:socket_init] 0-management: using system polling thread
>[2014-01-20 05:56:51.174737] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 05:56:51.496008] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
>[2014-01-20 05:56:52.340737] I [glusterd-handler.c:2346:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:56:52.340854] I [glusterd-handler.c:2398:__glusterd_handle_probe_query] 0-glusterd: Responded to 192.168.122.72, op_ret: 0, op_errno: 0, ret: 0
>[2014-01-20 05:56:54.029335] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:56:54.029493] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-2 (0), ret: 0
>[2014-01-20 05:56:54.829228] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 05:56:55.429464] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:56:55.429519] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:56:55.429529] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:56:55.429676] I [glusterd-rpc-ops.c:234:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 05:56:56.529377] I [glusterd-rpc-ops.c:306:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-01-20 05:56:56.529538] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:56:56.529577] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3, port: 0
>[2014-01-20 05:56:57.363173] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-4 24007
>[2014-01-20 05:56:57.364926] I [glusterd-handler.c:2931:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: glfs35-4 (24007)
>[2014-01-20 05:56:57.940659] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-01-20 05:56:57.940746] I [socket.c:3533:socket_init] 0-management: SSL support is NOT enabled
>[2014-01-20 05:56:57.940756] I [socket.c:3548:socket_init] 0-management: using system polling thread
>[2014-01-20 05:56:57.944601] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 05:56:58.329692] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
>[2014-01-20 05:56:59.207735] I [glusterd-handler.c:2346:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:56:59.207827] I [glusterd-handler.c:2398:__glusterd_handle_probe_query] 0-glusterd: Responded to 192.168.122.73, op_ret: 0, op_errno: 0, ret: 0
>[2014-01-20 05:57:00.751774] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:00.751884] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-3 (0), ret: 0
>[2014-01-20 05:57:01.507406] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 05:57:01.507475] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 05:57:02.496341] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:02.496428] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:02.496444] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:02.496525] I [glusterd-rpc-ops.c:234:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4
>[2014-01-20 05:57:03.563014] I [glusterd-rpc-ops.c:306:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-01-20 05:57:03.563146] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4, port: 0
>[2014-01-20 05:57:04.585794] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:57:04.585858] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:04.585978] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:04.586014] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:04.586029] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:04.586044] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 05:57:04.587440] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 05:57:06.152470] I [glusterd-handler.c:2346:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:06.152556] I [glusterd-handler.c:2398:__glusterd_handle_probe_query] 0-glusterd: Responded to 192.168.122.74, op_ret: 0, op_errno: 0, ret: 0
>[2014-01-20 05:57:06.775873] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:57:06.775930] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:06.775942] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:06.775953] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 05:57:06.863550] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:06.863655] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-4 (0), ret: 0
>[2014-01-20 05:57:07.885299] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 05:57:07.885360] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 05:57:07.885370] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4
>[2014-01-20 05:57:08.985770] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:08.985834] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:08.985852] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:09.068022] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:57:09.168002] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:10.034067] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:12.297668] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 05:57:12.297835] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:12.297852] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:12.297866] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 05:57:12.297882] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 05:57:13.019682] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 05:57:13.019733] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:13.019746] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:13.019759] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 05:57:13.019767] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 05:57:13.731028] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:13.731087] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:13.731098] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:13.731105] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 05:57:13.731112] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 05:57:16.519532] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 05:57:16.519596] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 05:57:16.519644] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 05:57:16.519674] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 05:57:16.519690] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 06:10:03.951205] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-2 24007
>[2014-01-20 06:10:03.973811] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-3 24007
>[2014-01-20 06:10:03.980593] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-4 24007
>[2014-01-20 06:12:27.711447] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-2 24007
>[2014-01-20 06:12:27.732483] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-3 24007
>[2014-01-20 06:12:27.740235] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glfs35-4 24007
>[2014-01-20 06:14:05.720042] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1 on port 49152
>[2014-01-20 06:14:05.721674] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-01-20 06:14:05.721730] I [socket.c:3533:socket_init] 0-management: SSL support is NOT enabled
>[2014-01-20 06:14:05.721740] I [socket.c:3548:socket_init] 0-management: using system polling thread
>[2014-01-20 06:14:11.710015] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-01-20 06:14:11.710094] I [socket.c:3533:socket_init] 0-management: SSL support is NOT enabled
>[2014-01-20 06:14:11.710105] I [socket.c:3548:socket_init] 0-management: using system polling thread
>[2014-01-20 06:14:11.724850] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
>[2014-01-20 07:43:59.390862] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 07:44:07.236334] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 07:44:18.394912] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 07:44:18.401623] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 07:44:25.777526] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 07:44:25.779239] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 07:44:25.791490] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 07:44:25.791534] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 07:45:34.665358] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 07:45:34.666847] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 07:45:34.670910] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 07:45:34.670945] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 07:45:45.093275] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 07:45:45.099017] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 07:46:36.078513] E [glusterd-volume-ops.c:1045:glusterd_op_stage_start_volume] 0-management: Volume myvol already started
>[2014-01-20 07:46:36.078564] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume myvol already started
>[2014-01-20 07:47:13.580554] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 07:47:13.581899] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 07:47:13.584962] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 07:47:13.584993] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 08:11:41.585805] W [socket.c:522:__socket_rwv] 0-management: readv on 192.168.122.73:24007 failed (Connection timed out)
>[2014-01-20 08:11:41.585942] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 08:11:46.607821] E [socket.c:2161:socket_connect_finish] 0-management: connection to 192.168.122.73:24007 failed (No route to host)
>[2014-01-20 09:12:38.942537] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 09:12:39.884760] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 09:12:39.884877] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-3 (0), ret: 0
>[2014-01-20 09:12:40.895166] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 09:12:40.895209] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 09:12:40.895223] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4
>[2014-01-20 09:12:42.139851] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 09:12:42.139909] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 09:12:42.462333] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 09:12:42.462391] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 09:12:42.462406] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 09:12:42.462418] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 09:12:42.462429] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 09:12:45.296206] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 09:12:45.296266] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 09:12:45.296285] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 09:12:45.296300] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 09:12:45.296314] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 09:12:46.406841] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 09:12:46.406896] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 09:12:46.406922] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 09:12:46.406933] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 09:12:46.406944] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 09:12:47.295986] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 09:12:47.296562] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 09:12:47.296590] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 09:12:47.296596] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 09:12:47.296603] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 09:12:48.051645] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 09:12:49.417870] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3, port: 0
>[2014-01-20 09:12:50.106745] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 09:12:50.106791] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 09:12:50.106799] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 09:12:50.106804] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 09:12:50.106810] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 09:15:58.895694] W [socket.c:522:__socket_rwv] 0-management: readv on 192.168.122.74:24007 failed (Connection timed out)
>[2014-01-20 09:15:58.895788] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 09:16:01.896789] E [socket.c:2161:socket_connect_finish] 0-management: connection to 192.168.122.74:24007 failed (No route to host)
>[2014-01-20 09:26:10.586259] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 09:26:10.587258] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 09:26:10.589103] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 09:26:10.589123] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 20:20:17.702181] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:20:17.702253] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:20:17.702270] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:20:17.702280] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:20:17.702291] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:20:17.846651] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 20:20:17.846707] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:20:17.846718] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:20:17.846730] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 20:20:17.846742] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:20:18.012224] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 20:20:22.074280] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:20:22.074373] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-4 (0), ret: 0
>[2014-01-20 20:20:23.423451] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 20:20:23.423493] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 20:20:23.423503] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4
>[2014-01-20 20:20:24.701181] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:20:24.701233] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 20:20:25.268843] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:20:26.557685] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:20:26.557730] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:20:26.557740] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:20:26.557747] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:20:26.557753] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 20:20:29.579070] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4, port: 0
>[2014-01-20 20:20:30.612733] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:20:30.612785] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:20:30.612798] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:20:30.612809] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:20:30.612820] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 20:24:18.346925] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:24:28.238485] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 20:24:28.239800] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 20:24:28.244958] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 20:24:28.245001] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 20:26:43.268971] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:26:44.783110] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:26:56.245784] W [socket.c:522:__socket_rwv] 0-management: readv on 192.168.122.73:24007 failed (Connection timed out)
>[2014-01-20 20:26:58.705055] E [socket.c:2161:socket_connect_finish] 0-management: connection to 192.168.122.73:24007 failed (No route to host)
>[2014-01-20 20:27:04.760475] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:27:04.761337] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:27:11.241063] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:27:31.942038] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:28:08.831450] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 3
>[2014-01-20 20:28:12.243694] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:28:12.243807] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glfs35-3 (0), ret: 0
>[2014-01-20 20:28:13.621139] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, host: glfs35-2
>[2014-01-20 20:28:13.621183] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3
>[2014-01-20 20:28:13.621199] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, host: glfs35-4
>[2014-01-20 20:28:14.987922] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:28:14.987973] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 20:28:14.987991] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:28:14.987997] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:28:14.988003] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 20:28:14.988013] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:28:14.988066] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, host: glfs35-3, port: 0
>[2014-01-20 20:28:15.765696] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab
>[2014-01-20 20:28:15.765781] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:28:15.765806] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:28:15.765827] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:28:15.765832] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:28:15.765838] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:28:16.888246] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f
>[2014-01-20 20:28:16.888301] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:28:16.888313] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:28:16.888324] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:28:16.888334] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036, hostname:glfs35-3
>[2014-01-20 20:28:18.143880] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:28:19.155052] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:28:19.155116] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:28:19.155129] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:28:19.155138] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:28:19.155148] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:28:19.681676] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:28:20.978794] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c7a936d8-5a8c-417b-bc22-13b13f4a4036
>[2014-01-20 20:28:20.978856] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 5a4c12e6-2c8f-425f-893b-ffeaec19d2ab, hostname:glfs35-2
>[2014-01-20 20:28:20.978875] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, hostname:192.168.122.71
>[2014-01-20 20:28:20.978887] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-01-20 20:28:20.978900] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: a1330f88-5e52-4f11-9721-ed32eed66c4f, hostname:glfs35-4
>[2014-01-20 20:28:26.117788] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 20:28:26.128848] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume myvol
>[2014-01-20 20:28:26.143965] W [glusterd-op-sm.c:3312:glusterd_op_modify_op_ctx] 0-management: Failed uuid to hostname conversion
>[2014-01-20 20:28:26.144003] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>[2014-01-20 20:28:30.361094] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:28:30.361585] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:29:49.008068] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:29:49.009675] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:29:58.129190] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:29:58.129233] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:30:32.056250] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:30:32.056291] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:32:44.995395] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:32:44.995439] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:34:08.451358] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:34:08.451377] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:34:23.140526] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:34:23.141391] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:34:29.260201] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:34:34.034623] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:37:04.939278] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:37:04.939303] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:38:15.494960] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:38:15.494988] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock
>[2014-01-20 20:38:46.881133] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(1)) xid = 0x4c sent = 2014-01-20 20:28:44.697259. timeout = 600 for 192.168.122.74:24007
>[2014-01-20 20:38:46.881207] E [glusterd-syncop.c:101:gd_collate_errors] 0-: Locking failed on glfs35-4. Please check log file for details.
>[2014-01-20 20:38:46.881312] E [glusterd-syncop.c:863:gd_lock_op_phase] 0-management: Failed to acquire lock
>[2014-01-20 20:38:46.882306] I [socket.c:3134:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-01-20 20:38:46.882318] E [rpcsvc.c:1206:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x5, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>[2014-01-20 20:38:46.882330] E [glusterd-utils.c:407:glusterd_submit_reply] 0-: Reply submission failed
>[2014-01-20 20:39:01.369838] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:41:08.614377] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-01-20 20:42:51.455959] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:42:51.457823] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>[2014-01-20 20:42:55.039906] E [glusterd-utils.c:153:glusterd_lock] 0-management: Unable to get lock for uuid: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5, lock held by: 942ef92f-20b2-49bb-8abf-1bdc2c2112e5
>[2014-01-20 20:42:55.039927] E [glusterd-syncop.c:1218:gd_sync_task_begin] 0-management: Unable to acquire lock

Actions: View

Attachments on bug 1055747: 852843