Attachment 892840 Details for Bug 1094716 – glusterd log

glusterd log

etc-glusterfs-glusterd.vol.log (text/x-log), 29.85 KB, created by Sachidananda Urs on 2014-05-06 10:57:22 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Sachidananda Urs

Created: 2014-05-06 10:57:22 UTC

Size: 29.85 KB

patch

obsolete

>[2014-05-06 03:16:34.812140] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.37.132:24007 failed (No data available)
>[2014-05-06 03:17:48.041619] E [socket.c:2158:socket_connect_finish] 0-management: connection to 10.70.37.132:24007 failed (Connection timed out)
>[2014-05-06 03:33:20.718393] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 03:33:20.719124] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 03:33:20.719168] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 03:33:20.918636] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.132, port: 24007
>[2014-05-06 03:35:14.208211] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.37.132:24007 failed (No data available)
>[2014-05-06 03:36:27.754558] E [socket.c:2158:socket_connect_finish] 0-management: connection to 10.70.37.132:24007 failed (Connection timed out)
>[2014-05-06 03:36:39.213268] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.132, port: 24007
>[2014-05-06 03:37:16.049060] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.37.132:24007 failed (No data available)
>[2014-05-06 03:37:26.789485] E [socket.c:2158:socket_connect_finish] 0-management: connection to 10.70.37.132:24007 failed (Connection refused)
>[2014-05-06 03:48:30.751798] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.132, port: 24007
>[2014-05-06 03:48:45.344513] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-05-06 03:48:58.827867] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 03:48:58.827963] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 03:48:58.828818] I [glusterd-handshake.c:562:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
>[2014-05-06 03:48:58.834657] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 03:48:58.834702] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 03:48:58.835179] I [glusterd-handler.c:2324:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 03:48:58.835295] I [glusterd-handler.c:2376:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.37.132, op_ret: 0, op_errno: 0, ret: 0
>[2014-05-06 03:48:58.853293] I [glusterd-handler.c:2028:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 03:48:58.853427] I [glusterd-handler.c:3059:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.37.132 (0), ret: 0
>[2014-05-06 03:48:59.039562] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 03:48:59.039610] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 03:49:20.752947] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>[2014-05-06 03:49:20.902876] I [glusterd-rpc-ops.c:467:__glusterd_friend_remove_cbk] 0-glusterd: Received ACC from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.173, port: 0
>[2014-05-06 03:49:20.957705] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=2236 max=4 total=80
>[2014-05-06 03:49:20.957733] I [mem-pool.c:539:mem_pool_destroy] 0-management: size=124 max=4 total=80
>[2014-05-06 03:49:20.957826] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 03:49:20.957898] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 03:49:26.052369] I [glusterd-handler.c:821:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.70.37.132 24007
>[2014-05-06 03:49:26.157478] I [glusterd-handler.c:2905:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.70.37.132 (24007)
>[2014-05-06 03:49:26.166111] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 03:49:26.173119] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 03:49:26.173163] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 03:49:26.178922] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
>[2014-05-06 03:49:26.296282] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.132
>[2014-05-06 03:49:26.305287] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-05-06 03:49:26.305641] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 03:49:26.305676] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 03:49:26.306862] I [glusterd-handshake.c:562:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
>[2014-05-06 03:49:26.312037] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 03:49:26.312077] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 03:49:55.574075] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-05-06 03:50:01.251678] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>[2014-05-06 03:50:11.427720] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>[2014-05-06 03:59:27.218841] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x5x sent = 2014-05-06 03:49:26.296729. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 03:59:27.218988] I [socket.c:3106:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-05-06 03:59:27.219010] E [rpcsvc.c:1204:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 1) to rpc-transport (socket.management)
>[2014-05-06 03:59:27.219036] E [glusterd-utils.c:405:glusterd_submit_reply] 0-: Reply submission failed
>[2014-05-06 04:00:07.226607] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(3)) xid = 0x6x sent = 2014-05-06 03:50:01.251846. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 04:00:07.226785] I [socket.c:3106:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-05-06 04:00:07.226806] E [rpcsvc.c:1204:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 2) to rpc-transport (socket.management)
>[2014-05-06 04:00:07.226832] E [glusterd-utils.c:405:glusterd_submit_reply] 0-: Reply submission failed
>[2014-05-06 04:00:07.231707] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 04:00:07.237868] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 04:00:07.251485] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 04:00:17.228955] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(3)) xid = 0x7x sent = 2014-05-06 03:50:11.427817. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 04:00:17.229110] I [socket.c:3106:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-05-06 04:00:17.229129] E [rpcsvc.c:1204:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 2) to rpc-transport (socket.management)
>[2014-05-06 04:00:17.229155] E [glusterd-utils.c:405:glusterd_submit_reply] 0-: Reply submission failed
>[2014-05-06 04:00:17.232068] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 04:00:17.232133] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 04:00:17.232165] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 04:55:01.432955] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>pending frames:
>frame : type(0) op(0)
>
>patchset: git://git.gluster.com/glusterfs.git
>signal received: 6
>time of crash: 2014-05-06 04:55:01configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>fdatasync 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 3.4.0.59rhs
>/lib64/libc.so.6[0x3dbce32960]
>/lib64/libc.so.6(gsignal+0x35)[0x3dbce328e5]
>/lib64/libc.so.6(abort+0x175)[0x3dbce340c5]
>/lib64/libc.so.6[0x3dbce707f7]
>/lib64/libc.so.6[0x3dbce76126]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_peer_destroy+0x3f)[0x7ffd358e4e5f]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_friend_cleanup+0xb8)[0x7ffd358ee818]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(+0x37c2f)[0x7ffd358d0c2f]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x19e)[0x7ffd358d152e]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(__glusterd_handle_cli_list_friends+0xba)[0x7ffd358cee9a]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7ffd358bf78f]
>/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7ffd39728192]
>/lib64/libc.so.6[0x3dbce43bb0]
>---------
>[2014-05-06 05:13:07.056356] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.0.59rhs (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-05-06 05:13:07.063090] I [glusterd.c:1150:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-05-06 05:13:07.066815] I [socket.c:3505:socket_init] 0-socket.management: SSL support is NOT enabled
>[2014-05-06 05:13:07.066837] I [socket.c:3520:socket_init] 0-socket.management: using system polling thread
>[2014-05-06 05:13:07.107678] C [rdma.c:4099:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
>[2014-05-06 05:13:07.107858] E [rdma.c:4990:init] 0-rdma.management: Failed to initialize IB Device
>[2014-05-06 05:13:07.107896] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
>[2014-05-06 05:13:07.107939] W [rpcsvc.c:1481:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-05-06 05:13:07.108042] I [socket.c:3505:socket_init] 0-socket.management: SSL support is NOT enabled
>[2014-05-06 05:13:07.108081] I [socket.c:3520:socket_init] 0-socket.management: using system polling thread
>[2014-05-06 05:13:12.525048] I [glusterd-store.c:1406:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>[2014-05-06 05:13:12.543258] E [glusterd-store.c:1957:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
>[2014-05-06 05:13:12.543334] E [glusterd-store.c:1957:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
>[2014-05-06 05:13:12.543382] E [glusterd-store.c:1957:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
>[2014-05-06 05:13:12.543405] E [glusterd-store.c:1957:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
>[2014-05-06 05:13:12.821510] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
>[2014-05-06 05:13:12.823638] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
>[2014-05-06 05:13:12.827324] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
>[2014-05-06 05:13:12.827502] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:12.827611] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:12.827631] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:12.834190] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:12.834363] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:12.834407] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:12.838212] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:12.838267] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:12.838287] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:12.844509] I [glusterd.c:140:glusterd_uuid_init] 0-management: retrieved UUID: dc0eee09-b5d5-454e-8431-f220e27ae1f2
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13: 
>+------------------------------------------------------------------------------+
>[2014-05-06 05:13:12.849287] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:12.849558] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:12.857258] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:12.857555] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:12.857728] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:12.858052] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 05:13:12.858084] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 05:13:12.862017] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /rhs/brick1/r0 on port 49152
>[2014-05-06 05:13:12.863209] I [glusterd-handshake.c:562:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
>[2014-05-06 05:13:13.367637] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193, host: 10.70.37.208, port: 0
>[2014-05-06 05:13:13.469885] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:13.470006] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:13.470027] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:14.474348] E [glusterd-utils.c:4007:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/3a37185558a28c0892ab1a139f0872c4.socket error: Permission denied
>[2014-05-06 05:13:14.534667] I [glusterd-utils.c:4041:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-05-06 05:13:14.535041] I [glusterd-utils.c:4046:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-05-06 05:13:14.535308] I [glusterd-utils.c:4051:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-05-06 05:13:14.535593] I [glusterd-utils.c:4056:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-05-06 05:13:14.535725] I [glusterd-utils.c:4061:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-05-06 05:13:14.535874] I [glusterd-utils.c:4066:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-05-06 05:13:14.539903] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:14.539964] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:14.539975] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:15.540975] E [glusterd-utils.c:4007:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/7a78a9c983ab0848fa73cdf4c29ebe73.socket error: No such file or directory
>[2014-05-06 05:13:15.546735] I [rpc-clnt.c:977:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-05-06 05:13:15.546877] I [socket.c:3505:socket_init] 0-management: SSL support is NOT enabled
>[2014-05-06 05:13:15.546904] I [socket.c:3520:socket_init] 0-management: using system polling thread
>[2014-05-06 05:13:15.642895] I [glusterd-handler.c:2028:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:13:15.643071] I [glusterd-handler.c:3059:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.37.69 (0), ret: 0
>[2014-05-06 05:13:15.735816] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 4525556f-9f49-4efd-9cba-863e6bf36193, host: 10.70.37.208
>[2014-05-06 05:13:15.735870] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 301a8582-2038-4385-ad4f-96187d67f683, host: 10.70.37.69
>[2014-05-06 05:13:15.867577] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:13:15.867680] I [glusterd-handler.c:2190:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:13:15.867724] I [glusterd-handler.c:2235:__glusterd_handle_friend_update] 0-: Received uuid: 301a8582-2038-4385-ad4f-96187d67f683, hostname:10.70.37.69
>[2014-05-06 05:13:15.867746] I [glusterd-handler.c:2235:__glusterd_handle_friend_update] 0-: Received uuid: dc0eee09-b5d5-454e-8431-f220e27ae1f2, hostname:10.70.37.173
>[2014-05-06 05:13:15.867761] I [glusterd-handler.c:2244:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-05-06 05:13:15.867890] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683, host: 10.70.37.69, port: 0
>[2014-05-06 05:13:15.873950] I [glusterd-handler.c:2190:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:13:15.874002] I [glusterd-handler.c:2235:__glusterd_handle_friend_update] 0-: Received uuid: 4525556f-9f49-4efd-9cba-863e6bf36193, hostname:10.70.37.208
>[2014-05-06 05:13:15.874023] I [glusterd-handler.c:2235:__glusterd_handle_friend_update] 0-: Received uuid: dc0eee09-b5d5-454e-8431-f220e27ae1f2, hostname:10.70.37.173
>[2014-05-06 05:13:15.874038] I [glusterd-handler.c:2244:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-05-06 05:13:15.874129] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:15.874154] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:13:15.876720] I [socket.c:2235:socket_event_handler] 0-transport: disconnecting now
>[2014-05-06 05:13:15.876881] I [socket.c:2235:socket_event_handler] 0-transport: disconnecting now
>[2014-05-06 05:13:15.877032] I [glusterd-handshake.c:562:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
>[2014-05-06 05:13:15.881864] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:13:15.881931] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8, host: 10.70.37.132
>[2014-05-06 05:13:15.882150] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:15.951871] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-05-06 05:13:15.952033] I [glusterd-handshake.c:562:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
>[2014-05-06 05:13:15.973283] I [glusterd-handler.c:2028:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:13:15.973390] I [glusterd-handler.c:3059:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.37.208 (0), ret: 0
>[2014-05-06 05:13:16.140312] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 4525556f-9f49-4efd-9cba-863e6bf36193, host: 10.70.37.208
>[2014-05-06 05:13:16.140341] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 301a8582-2038-4385-ad4f-96187d67f683, host: 10.70.37.69
>[2014-05-06 05:13:16.204687] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:16.204708] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:13:16.204788] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 05:13:16.204831] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 05:13:16.205045] I [glusterd-handler.c:2324:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 05:13:16.205093] I [glusterd-handler.c:2376:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.37.132, op_ret: 0, op_errno: 0, ret: 0
>[2014-05-06 05:13:16.205121] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:13:16.205140] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:13:16.205731] I [glusterd-handler.c:2028:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 05:13:16.285918] E [rpcsvc.c:226:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 2 in GF-DUMP
>[2014-05-06 05:13:16.285946] E [rpcsvc.c:493:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>[2014-05-06 05:13:23.836573] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:23.836629] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:45.223888] I [glusterd-handler.c:1018:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>[2014-05-06 05:13:52.337859] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>[2014-05-06 05:13:52.338054] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:52.348581] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:13:52.348616] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:16:21.100546] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>[2014-05-06 05:16:21.100737] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:16:21.110698] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:16:21.110740] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:23:23.961259] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x6x sent = 2014-05-06 05:13:15.882146. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 05:23:53.966939] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(3)) xid = 0x8x sent = 2014-05-06 05:13:52.338045. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 05:23:53.967088] I [socket.c:3106:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-05-06 05:23:53.967109] E [rpcsvc.c:1204:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 2) to rpc-transport (socket.management)
>[2014-05-06 05:23:53.967137] E [glusterd-utils.c:405:glusterd_submit_reply] 0-: Reply submission failed
>[2014-05-06 05:23:53.967308] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:23:53.967424] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:23:53.967539] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:23:53.972033] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 05:23:53.972258] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:23:53.972287] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:23:53.972324] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:23:53.972407] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:23:53.972442] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:23:53.972476] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:24:04.969705] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:24:04.969768] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.995763] E [rpc-clnt.c:208:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(3)) xid = 0x10x sent = 2014-05-06 05:16:21.100731. timeout = 600 for 10.70.37.132:24007
>[2014-05-06 05:26:23.995905] I [socket.c:3106:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>[2014-05-06 05:26:23.995926] E [rpcsvc.c:1204:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 2) to rpc-transport (socket.management)
>[2014-05-06 05:26:23.995957] E [glusterd-utils.c:405:glusterd_submit_reply] 0-: Reply submission failed
>[2014-05-06 05:26:23.996073] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.996230] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.996408] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.999191] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: da81a488-9e1b-4c2e-bb74-76ff5abe91a8
>[2014-05-06 05:26:23.999360] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4525556f-9f49-4efd-9cba-863e6bf36193
>[2014-05-06 05:26:23.999407] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 301a8582-2038-4385-ad4f-96187d67f683
>[2014-05-06 05:26:23.999446] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.999465] W [glusterd-ping.c:307:glusterd_ping_cbk] 0-management: socket or ib related error
>[2014-05-06 05:26:23.999532] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:23.999567] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:34.998771] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:26:34.998843] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-05-06 05:35:18.559243] I [glusterd-handler.c:916:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req
>pending frames:
>frame : type(0) op(0)
>
>patchset: git://git.gluster.com/glusterfs.git
>signal received: 6
>time of crash: 2014-05-06 05:35:18configuration details:
>argp 1
>backtrace 1
>dlfcn 1
>fdatasync 1
>libpthread 1
>llistxattr 1
>setfsid 1
>spinlock 1
>epoll.h 1
>xattr.h 1
>st_atim.tv_nsec 1
>package-string: glusterfs 3.4.0.59rhs
>/lib64/libc.so.6[0x3dbce32960]
>/lib64/libc.so.6(gsignal+0x35)[0x3dbce328e5]
>/lib64/libc.so.6(abort+0x175)[0x3dbce340c5]
>/lib64/libc.so.6[0x3dbce707f7]
>/lib64/libc.so.6[0x3dbce76126]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_peer_destroy+0x3f)[0x7f628f83ae5f]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_friend_cleanup+0xb8)[0x7f628f844818]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(+0x37c2f)[0x7f628f826c2f]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x19e)[0x7f628f82752e]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(__glusterd_handle_cli_deprobe+0x1a8)[0x7f628f825378]
>/usr/lib64/glusterfs/3.4.0.59rhs/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7f628f81578f]
>/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x37db04b192]
>/lib64/libc.so.6[0x3dbce43bb0]
>---------

Actions: View

Attachments on bug 1094716: 892840