Attachment 884819 Details for Bug 1083668 – Glusterd Logs

Glusterd Logs

etc-glusterfs-glusterd.vol.log (text/x-log), 66.64 KB, created by Lalatendu Mohanty on 2014-04-10 09:20:29 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2014-04-10 09:20:29 UTC

Size: 66.64 KB

patch

obsolete

>[2014-04-09 08:08:03.996653] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-glusterd: Started running glusterd version 3.5qa2 (args: glusterd --xlator-option *.upgrade=on -N)
>[2014-04-09 08:08:03.998943] I [graph.c:254:gf_add_cmdline_options] 0-management: adding option 'upgrade' for volume 'management' with value 'on'
>[2014-04-09 08:08:03.998992] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-04-09 08:08:04.002383] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5qa2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
>[2014-04-09 08:08:04.002401] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
>[2014-04-09 08:08:04.002411] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-04-09 08:08:04.003145] I [glusterd.c:372:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
>[2014-04-09 08:08:04.003430] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 08:08:04.003448] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 08:08:04.003456] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4
>[2014-04-09 08:08:04.006341] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory)
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option upgrade on
>  8:     option transport.socket.read-fail-log off
>  9:     option transport.socket.keepalive-interval 2
> 10:     option transport.socket.keepalive-time 10
> 11:     option transport-type rdma
> 12:     option working-directory /var/lib/glusterd
> 13: end-volume
> 14:  
>+------------------------------------------------------------------------------+
>[2014-04-09 08:08:04.026205] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down
>[2014-04-09 08:18:10.449978] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-04-09 08:18:10.460771] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-04-09 08:18:10.468933] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
>[2014-04-09 08:18:10.468971] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
>[2014-04-09 08:18:10.468988] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
>[2014-04-09 08:18:10.469075] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-04-09 08:18:16.557949] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 08:18:16.558018] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 08:18:16.558038] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13:  
>+------------------------------------------------------------------------------+
>[2014-04-09 08:18:52.850535] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4
>[2014-04-09 08:18:52.850653] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 08:18:52.850739] I [glusterd.c:173:glusterd_uuid_generate_save] 0-management: generated UUID: 1e3be0ed-0f3f-4364-a365-5181d15c3d16
>[2014-04-09 08:18:52.877063] I [glusterd-handler.c:2754:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae
>[2014-04-09 08:18:52.879664] I [glusterd-handler.c:2782:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.70.40.131 (24007)
>[2014-04-09 08:18:52.912630] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 08:18:52.920014] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0
>[2014-04-09 08:18:52.920226] I [glusterd-handler.c:2806:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.40.131, op_ret: 0, op_errno: 0, ret: 0
>[2014-04-09 08:18:52.921247] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae
>[2014-04-09 08:18:52.921485] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0
>[2014-04-09 08:18:53.085610] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 08:18:53.086796] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 08:18:53.121357] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131
>[2014-04-09 08:18:53.177812] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-04-09 08:18:53.177898] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131, port: 0
>[2014-04-09 08:18:53.177939] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131
>[2014-04-09 08:18:53.259049] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae
>[2014-04-09 08:18:53.259129] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae
>[2014-04-09 08:18:53.259168] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 1e3be0ed-0f3f-4364-a365-5181d15c3d16, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 08:18:53.259186] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 08:19:03.922306] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 08:19:26.698362] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/testvol_brick1 on port 49152
>[2014-04-09 08:19:26.701010] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 08:19:26.769177] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 08:19:26.774391] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 08:19:26.781066] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 08:19:26.781168] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 08:19:26.798095] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=testvol --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 08:19:26.960896] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=testvol --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:14:46.090140] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:14:46.095702] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=testvol --last=yes
>[2014-04-09 09:14:46.135813] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=testvol --last=yes
>[2014-04-09 09:14:46.135922] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:14:46.135994] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/5b1f4b67d531e9bb773f686a8f0da76d.socket error: No such file or directory
>[2014-04-09 09:14:46.178790] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f4b43725ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f4b437255f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:14:47.180762] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/92ecf1ce703775a9877d79643b07a3fd.socket error: Permission denied
>[2014-04-09 09:14:47.181392] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-04-09 09:14:47.181540] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-04-09 09:14:47.181701] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-04-09 09:14:47.181831] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-04-09 09:14:47.181964] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-04-09 09:14:47.182111] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-04-09 09:14:47.182257] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f4b43725ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f4b437255f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:14:48.182784] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/aec20b542745f6bdc6e288b06cbaa34d.socket error: No such file or directory
>[2014-04-09 09:14:48.183165] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7f4b4dbc058f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x7f4b4dbbef9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:14:48.186930] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/testvol_brick1 on port 49152
>[2014-04-09 09:15:04.253493] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.40.131:24007 failed (No data available)
>[2014-04-09 09:15:08.665129] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down
>[2014-04-09 09:21:03.102882] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-04-09 09:21:03.110642] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-04-09 09:21:03.118780] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
>[2014-04-09 09:21:03.118820] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
>[2014-04-09 09:21:03.118836] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
>[2014-04-09 09:21:03.118922] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-04-09 09:21:09.242325] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 09:21:09.242395] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 09:21:09.242416] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4
>[2014-04-09 09:21:09.242699] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory)
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13:  
>+------------------------------------------------------------------------------+
>[2014-04-09 09:21:38.460611] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4
>[2014-04-09 09:21:38.460700] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 09:21:38.460771] I [glusterd.c:173:glusterd_uuid_generate_save] 0-management: generated UUID: 51dc2165-b320-4a27-a602-86580d5e9be5
>[2014-04-09 09:21:38.488121] I [glusterd-handler.c:2754:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:21:38.491645] I [glusterd-handler.c:2782:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.70.40.131 (24007)
>[2014-04-09 09:21:38.540877] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:38.548055] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0
>[2014-04-09 09:21:38.548595] I [glusterd-handler.c:2806:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.40.131, op_ret: 0, op_errno: 0, ret: 0
>[2014-04-09 09:21:38.549377] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:21:38.549664] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0
>[2014-04-09 09:21:38.707528] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:21:38.708665] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:21:38.741027] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131
>[2014-04-09 09:21:38.797600] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>[2014-04-09 09:21:38.797807] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131, port: 0
>[2014-04-09 09:21:38.797893] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131
>[2014-04-09 09:21:38.864658] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:21:38.864722] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 09:21:38.864742] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 09:21:38.867869] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:21:46.024164] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:21:46.030887] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:46.077146] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:21:46.084270] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:46.119413] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:21:46.126710] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:46.245868] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:46.249771] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:21:46.255609] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:21:46.255687] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:21:46.274141] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:21:46.408784] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:21:49.550248] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:21:55.690481] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:21:55.690614] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:21:55.690709] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:21.221039] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:21.221199] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:21.221254] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:21.227926] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:22:21.241973] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes
>[2014-04-09 09:22:21.302051] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes
>[2014-04-09 09:22:21.302621] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:21.302680] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:21.302817] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory
>[2014-04-09 09:22:21.302867] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:21.302922] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory
>[2014-04-09 09:22:21.303065] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:21.303101] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:21.303161] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory
>[2014-04-09 09:22:21.392623] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:22.393145] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied
>[2014-04-09 09:22:22.393626] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-04-09 09:22:22.393950] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-04-09 09:22:22.394231] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-04-09 09:22:22.394546] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-04-09 09:22:22.394824] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-04-09 09:22:22.395109] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-04-09 09:22:22.395360] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:23.395763] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory
>[2014-04-09 09:22:23.396569] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:23.396818] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:22:23.401558] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:22:29.321511] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:22:29.324699] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:29.365199] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:22:29.370017] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:29.410152] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:22:29.416890] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:29.580172] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:29.588998] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:29.596199] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:22:29.596287] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:22:29.612844] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:22:29.771129] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:22:37.207177] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:37.207361] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:37.207474] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:45.721076] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:45.721247] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:45.721304] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:22:45.722916] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:22:45.728314] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket failed (No data available)
>[2014-04-09 09:22:45.733896] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1
>[2014-04-09 09:22:45.734000] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:22:45.746974] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes
>[2014-04-09 09:22:45.806387] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes
>[2014-04-09 09:22:45.806671] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:45.806720] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:45.806823] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory
>[2014-04-09 09:22:45.806954] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:45.806989] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:45.807066] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory
>[2014-04-09 09:22:45.807128] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:22:45.807184] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory
>[2014-04-09 09:22:45.902088] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:46.902738] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied
>[2014-04-09 09:22:46.903392] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-04-09 09:22:46.903735] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-04-09 09:22:46.904051] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-04-09 09:22:46.904339] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-04-09 09:22:46.904655] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-04-09 09:22:46.904935] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-04-09 09:22:46.905186] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:47.906655] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory
>[2014-04-09 09:22:47.907292] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:22:47.907639] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:22:54.017430] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:22:54.020605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:54.060518] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:22:54.066052] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:54.117782] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:22:54.127677] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:54.264914] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:54.269411] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:22:54.276411] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:22:54.276682] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:22:54.302214] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:22:54.440442] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:23:00.717877] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:00.718083] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:00.718163] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:04.284065] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=on
>[2014-04-09 09:23:08.935601] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=on
>[2014-04-09 09:23:10.068692] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:10.068839] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:10.068907] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:20.111697] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=on
>[2014-04-09 09:23:25.148659] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=on
>[2014-04-09 09:23:26.176291] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:26.176413] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:26.176475] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:38.370765] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:38.370887] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:38.370977] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:23:42.044575] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=off
>[2014-04-09 09:23:43.062477] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=off
>[2014-04-09 09:25:37.980757] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:37.980905] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:37.980964] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:37.982821] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:25:37.989910] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket failed (No data available)
>[2014-04-09 09:25:37.995868] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1
>[2014-04-09 09:25:37.996006] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:25:38.000805] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick3
>[2014-04-09 09:25:38.008975] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes
>[2014-04-09 09:25:38.068072] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes
>[2014-04-09 09:25:38.068345] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:25:38.068398] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:25:38.068539] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory
>[2014-04-09 09:25:38.068669] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:25:38.068704] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:25:38.068763] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory
>[2014-04-09 09:25:38.068804] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped
>[2014-04-09 09:25:38.068854] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory
>[2014-04-09 09:25:38.165119] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:25:39.165659] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied
>[2014-04-09 09:25:39.166253] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-04-09 09:25:39.166582] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-04-09 09:25:39.166863] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-04-09 09:25:39.167159] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-04-09 09:25:39.167432] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-04-09 09:25:39.167736] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-04-09 09:25:39.167980] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:25:40.168408] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory
>[2014-04-09 09:25:40.169232] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:25:40.169319] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:25:40.174638] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:25:40.987674] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152
>[2014-04-09 09:25:40.990819] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:41.030406] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153
>[2014-04-09 09:25:41.036075] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:41.076219] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154
>[2014-04-09 09:25:41.083389] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:41.220013] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:41.224322] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:41.232363] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:25:41.232923] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:25:41.234948] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:25:41.372776] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:25:46.927453] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:46.928321] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:46.928804] E [glusterd-op-sm.c:6205:glusterd_op_sm] 0-management: lock failed due to Device or resource busy
>[2014-04-09 09:25:46.928934] E [glusterd-op-sm.c:6205:glusterd_op_sm] 0-management: lock failed due to Device or resource busy
>[2014-04-09 09:25:46.929017] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:48.414344] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:48.414462] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:48.414516] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:49.888768] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:49.888938] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:49.889038] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:51.157743] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:51.157848] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:51.157908] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:52.484470] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:52.484590] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:52.484648] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:53.963238] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:53.963406] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:53.963466] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:55.436341] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:55.436475] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:55.436533] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:59.345817] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:25:59.345956] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 09:25:59.345975] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 09:25:59.345993] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 09:25:59.392605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:25:59.401135] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0
>[2014-04-09 09:25:59.402356] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:59.403469] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:59.422855] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4
>[2014-04-09 09:25:59.428101] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com, port: 0
>[2014-04-09 09:25:59.434819] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 180d8cc5-4883-495f-870a-ea64514eca44
>[2014-04-09 09:25:59.435089] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to rhsauto022.lab.eng.blr.redhat.com (0), ret: 0
>[2014-04-09 09:25:59.461908] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131
>[2014-04-09 09:25:59.461979] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 09:25:59.462212] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:25:59.472978] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 09:25:59.473101] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44
>[2014-04-09 09:25:59.474419] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:01.005950] E [glusterd-brick-ops.c:1232:glusterd_op_stage_add_brick] 0-management: Unable to get replica count
>[2014-04-09 09:26:03.283641] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=hosdu --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd
>[2014-04-09 09:26:03.283806] I [glusterd-brick-ops.c:1023:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it
>[2014-04-09 09:26:03.490696] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:26:04.495701] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied
>[2014-04-09 09:26:04.496443] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
>[2014-04-09 09:26:04.496744] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
>[2014-04-09 09:26:04.497026] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
>[2014-04-09 09:26:04.497352] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
>[2014-04-09 09:26:04.497634] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
>[2014-04-09 09:26:04.497908] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
>[2014-04-09 09:26:04.504087] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:26:04.505557] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:26:05.506084] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory
>[2014-04-09 09:26:05.515369] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:26:05.517014] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:26:05.517387] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 09:26:09.620528] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:09.620630] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:14.101851] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:14.101994] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:14.102073] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:15.641234] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:15.641419] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:15.641525] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:18.989750] W [glusterd-rebalance.c:570:glusterd_op_stage_rebalance] 0-management: Missing rebalance-id
>[2014-04-09 09:26:29.042577] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 09:26:29.610578] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:26:29.929640] I [glusterd-handshake.c:365:__server_event_notify] 0-: received defrag status updated
>[2014-04-09 09:26:29.929726] E [glusterd-rebalance.c:768:glusterd_defrag_event_notify_handle] 0-: Failed to update status
>[2014-04-09 09:26:29.936354] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/gluster/gluster-rebalance-95452d6c-d0e6-4934-b9ff-ae485d32048a.sock failed (No data available)
>[2014-04-09 09:26:29.948417] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(__glusterd_defrag_notify+0x2d2) [0x7f239d759b22] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 09:27:36.841904] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:27:36.842181] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:27:36.842313] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:27:37.501911] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:27:37.502107] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 09:27:37.502187] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:46:06.559127] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.40.131:24007 failed (No data available)
>[2014-04-09 10:46:06.559620] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held
>[2014-04-09 10:46:10.277844] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.36.252:24007 failed (No data available)
>[2014-04-09 10:46:10.278418] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held
>[2014-04-09 10:46:13.645331] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down
>[2014-04-09 10:52:44.685969] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-04-09 10:52:44.698572] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-04-09 10:52:44.706613] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
>[2014-04-09 10:52:44.706651] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
>[2014-04-09 10:52:44.706668] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
>[2014-04-09 10:52:44.706754] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-04-09 10:52:50.793857] I [glusterd-store.c:1428:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 4
>[2014-04-09 10:52:51.198271] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0
>[2014-04-09 10:52:51.199914] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0
>[2014-04-09 10:52:51.200034] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.206776] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.545960] I [glusterd.c:143:glusterd_uuid_init] 0-management: retrieved UUID: 51dc2165-b320-4a27-a602-86580d5e9be5
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13:  
>+------------------------------------------------------------------------------+
>[2014-04-09 10:52:51.562014] E [socket.c:2162:socket_connect_finish] 0-management: connection to 10.70.36.252:24007 failed (Connection refused)
>[2014-04-09 10:52:51.563330] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:52:51.564369] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4
>[2014-04-09 10:52:51.572939] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:52:51.574805] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 10:52:51.575094] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0
>[2014-04-09 10:52:51.647115] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131
>[2014-04-09 10:52:51.647198] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 10:52:51.725609] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.737533] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.749765] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.775698] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:51.796296] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:56.821605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2014-04-09 10:52:56.834913] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 10:52:56.839580] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1
>[2014-04-09 10:52:56.844625] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 10:52:56.849246] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick3
>[2014-04-09 10:52:56.854591] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 10:52:56.858762] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick5
>[2014-04-09 10:52:56.858835] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 10:52:56.858933] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now
>[2014-04-09 10:52:56.859088] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131, port: 0
>[2014-04-09 10:52:56.869072] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 10:52:56.869161] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 10:52:56.869186] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 10:52:56.869201] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 10:52:56.913386] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154
>[2014-04-09 10:52:56.931654] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153
>[2014-04-09 10:52:57.123262] I [glusterd-handshake.c:365:__server_event_notify] 0-: received defrag status updated
>[2014-04-09 10:52:57.123312] E [glusterd-rebalance.c:768:glusterd_defrag_event_notify_handle] 0-: Failed to update status
>[2014-04-09 10:52:57.135267] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/gluster/gluster-rebalance-95452d6c-d0e6-4934-b9ff-ae485d32048a.sock failed (No data available)
>[2014-04-09 10:52:57.144662] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(__glusterd_defrag_notify+0x2d2) [0x7f6c5ce51b22] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f6c5ce1e5f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this
>[2014-04-09 10:52:58.237991] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:01.486679] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:01.492173] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4
>[2014-04-09 10:53:01.589964] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 10:53:01.590091] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 10:53:01.590117] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 10:53:01.590330] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 10:53:02.258951] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:06.606095] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 180d8cc5-4883-495f-870a-ea64514eca44
>[2014-04-09 10:53:06.606506] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to rhsauto022.lab.eng.blr.redhat.com (0), ret: 0
>[2014-04-09 10:53:06.616070] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131
>[2014-04-09 10:53:06.616119] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com
>[2014-04-09 10:53:06.616361] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:06.646815] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77
>[2014-04-09 10:53:06.657091] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:06.668128] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com, port: 0
>[2014-04-09 10:53:06.688215] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 180d8cc5-4883-495f-870a-ea64514eca44
>[2014-04-09 10:53:06.688282] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com
>[2014-04-09 10:53:06.688301] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
>[2014-04-09 10:53:06.688320] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, hostname:10.70.40.131
>[2014-04-09 10:53:06.698172] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49152
>[2014-04-09 10:53:06.714191] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154
>[2014-04-09 10:53:06.720195] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153
>[2014-04-09 10:53:12.303712] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 10:53:17.333730] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s
>[2014-04-09 11:00:09.428152] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.36.252:24007 failed (No data available)
>[2014-04-09 11:00:09.428391] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held
>[2014-04-09 11:00:09.471205] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49152
>[2014-04-09 11:00:09.484356] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154
>[2014-04-09 11:00:09.491623] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153
>[2014-04-09 11:00:20.416661] E [socket.c:2162:socket_connect_finish] 0-management: connection to 10.70.36.252:24007 failed (Connection refused)
>[2014-04-09 11:00:26.181459] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down
>[2014-04-09 11:11:45.133901] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>[2014-04-09 11:11:45.139978] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory
>[2014-04-09 11:11:45.145965] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
>[2014-04-09 11:11:45.146002] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
>[2014-04-09 11:11:45.146018] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
>[2014-04-09 11:11:45.146109] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>[2014-04-09 11:11:48.009226] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 11:11:48.009277] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
>[2014-04-09 11:11:48.009288] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4
>[2014-04-09 11:11:48.009425] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory)
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option transport.socket.listen-backlog 128
>  7:     option transport.socket.read-fail-log off
>  8:     option transport.socket.keepalive-interval 2
>  9:     option transport.socket.keepalive-time 10
> 10:     option transport-type rdma
> 11:     option working-directory /var/lib/glusterd
> 12: end-volume
> 13:  
>+------------------------------------------------------------------------------+

Actions: View

Attachments on bug 1083668: 884818 | 884819 | 884820 | 889196