Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 884819 Details for
Bug 1083668
BVT: profile tests hang because of Gluster issues
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Glusterd Logs
etc-glusterfs-glusterd.vol.log (text/x-log), 66.64 KB, created by
Lalatendu Mohanty
on 2014-04-10 09:20:29 UTC
(
hide
)
Description:
Glusterd Logs
Filename:
MIME Type:
Creator:
Lalatendu Mohanty
Created:
2014-04-10 09:20:29 UTC
Size:
66.64 KB
patch
obsolete
>[2014-04-09 08:08:03.996653] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-glusterd: Started running glusterd version 3.5qa2 (args: glusterd --xlator-option *.upgrade=on -N) >[2014-04-09 08:08:03.998943] I [graph.c:254:gf_add_cmdline_options] 0-management: adding option 'upgrade' for volume 'management' with value 'on' >[2014-04-09 08:08:03.998992] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory >[2014-04-09 08:08:04.002383] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5qa2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory >[2014-04-09 08:08:04.002401] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine >[2014-04-09 08:08:04.002411] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >[2014-04-09 08:08:04.003145] I [glusterd.c:372:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system >[2014-04-09 08:08:04.003430] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 08:08:04.003448] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 08:08:04.003456] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4 >[2014-04-09 08:08:04.006341] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory) >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option upgrade on > 8: option transport.socket.read-fail-log off > 9: option transport.socket.keepalive-interval 2 > 10: option transport.socket.keepalive-time 10 > 11: option transport-type rdma > 12: option working-directory /var/lib/glusterd > 13: end-volume > 14: >+------------------------------------------------------------------------------+ >[2014-04-09 08:08:04.026205] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down >[2014-04-09 08:18:10.449978] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) >[2014-04-09 08:18:10.460771] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory >[2014-04-09 08:18:10.468933] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) >[2014-04-09 08:18:10.468971] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device >[2014-04-09 08:18:10.468988] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed >[2014-04-09 08:18:10.469075] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >[2014-04-09 08:18:16.557949] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 08:18:16.558018] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 08:18:16.558038] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4 >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: >+------------------------------------------------------------------------------+ >[2014-04-09 08:18:52.850535] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4 >[2014-04-09 08:18:52.850653] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 08:18:52.850739] I [glusterd.c:173:glusterd_uuid_generate_save] 0-management: generated UUID: 1e3be0ed-0f3f-4364-a365-5181d15c3d16 >[2014-04-09 08:18:52.877063] I [glusterd-handler.c:2754:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae >[2014-04-09 08:18:52.879664] I [glusterd-handler.c:2782:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.70.40.131 (24007) >[2014-04-09 08:18:52.912630] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 08:18:52.920014] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0 >[2014-04-09 08:18:52.920226] I [glusterd-handler.c:2806:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.40.131, op_ret: 0, op_errno: 0, ret: 0 >[2014-04-09 08:18:52.921247] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae >[2014-04-09 08:18:52.921485] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0 >[2014-04-09 08:18:53.085610] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 08:18:53.086796] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 08:18:53.121357] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131 >[2014-04-09 08:18:53.177812] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req >[2014-04-09 08:18:53.177898] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131, port: 0 >[2014-04-09 08:18:53.177939] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae, host: 10.70.40.131 >[2014-04-09 08:18:53.259049] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae >[2014-04-09 08:18:53.259129] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2a32756e-e177-4e34-b35c-1d6831f605ae >[2014-04-09 08:18:53.259168] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 1e3be0ed-0f3f-4364-a365-5181d15c3d16, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 08:18:53.259186] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 08:19:03.922306] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 08:19:26.698362] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/testvol_brick1 on port 49152 >[2014-04-09 08:19:26.701010] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 08:19:26.769177] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 08:19:26.774391] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 08:19:26.781066] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 08:19:26.781168] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 08:19:26.798095] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=testvol --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 08:19:26.960896] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=testvol --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:14:46.090140] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:14:46.095702] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=testvol --last=yes >[2014-04-09 09:14:46.135813] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=testvol --last=yes >[2014-04-09 09:14:46.135922] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:14:46.135994] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/5b1f4b67d531e9bb773f686a8f0da76d.socket error: No such file or directory >[2014-04-09 09:14:46.178790] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f4b43725ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f4b437255f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:14:47.180762] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/92ecf1ce703775a9877d79643b07a3fd.socket error: Permission denied >[2014-04-09 09:14:47.181392] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully >[2014-04-09 09:14:47.181540] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully >[2014-04-09 09:14:47.181701] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully >[2014-04-09 09:14:47.181831] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully >[2014-04-09 09:14:47.181964] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully >[2014-04-09 09:14:47.182111] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully >[2014-04-09 09:14:47.182257] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f4b43725ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f4b437255f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:14:48.182784] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/aec20b542745f6bdc6e288b06cbaa34d.socket error: No such file or directory >[2014-04-09 09:14:48.183165] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x7f4b4dbc058f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x7f4b4dbbef9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x7f4b4dbbdfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:14:48.186930] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/testvol_brick1 on port 49152 >[2014-04-09 09:15:04.253493] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.40.131:24007 failed (No data available) >[2014-04-09 09:15:08.665129] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down >[2014-04-09 09:21:03.102882] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) >[2014-04-09 09:21:03.110642] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory >[2014-04-09 09:21:03.118780] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) >[2014-04-09 09:21:03.118820] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device >[2014-04-09 09:21:03.118836] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed >[2014-04-09 09:21:03.118922] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >[2014-04-09 09:21:09.242325] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 09:21:09.242395] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 09:21:09.242416] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4 >[2014-04-09 09:21:09.242699] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory) >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: >+------------------------------------------------------------------------------+ >[2014-04-09 09:21:38.460611] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4 >[2014-04-09 09:21:38.460700] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 09:21:38.460771] I [glusterd.c:173:glusterd_uuid_generate_save] 0-management: generated UUID: 51dc2165-b320-4a27-a602-86580d5e9be5 >[2014-04-09 09:21:38.488121] I [glusterd-handler.c:2754:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:21:38.491645] I [glusterd-handler.c:2782:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.70.40.131 (24007) >[2014-04-09 09:21:38.540877] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:38.548055] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0 >[2014-04-09 09:21:38.548595] I [glusterd-handler.c:2806:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.70.40.131, op_ret: 0, op_errno: 0, ret: 0 >[2014-04-09 09:21:38.549377] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:21:38.549664] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0 >[2014-04-09 09:21:38.707528] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:21:38.708665] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:21:38.741027] I [glusterd-rpc-ops.c:235:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131 >[2014-04-09 09:21:38.797600] I [glusterd-rpc-ops.c:307:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req >[2014-04-09 09:21:38.797807] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131, port: 0 >[2014-04-09 09:21:38.797893] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131 >[2014-04-09 09:21:38.864658] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:21:38.864722] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 09:21:38.864742] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 09:21:38.867869] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:21:46.024164] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:21:46.030887] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:46.077146] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:21:46.084270] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:46.119413] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:21:46.126710] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:46.245868] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:46.249771] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:21:46.255609] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:21:46.255687] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:21:46.274141] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:21:46.408784] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:21:49.550248] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:21:55.690481] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:21:55.690614] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:21:55.690709] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:21.221039] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:21.221199] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:21.221254] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:21.227926] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:22:21.241973] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes >[2014-04-09 09:22:21.302051] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes >[2014-04-09 09:22:21.302621] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:21.302680] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:21.302817] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory >[2014-04-09 09:22:21.302867] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:21.302922] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory >[2014-04-09 09:22:21.303065] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:21.303101] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:21.303161] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory >[2014-04-09 09:22:21.392623] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:22.393145] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied >[2014-04-09 09:22:22.393626] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully >[2014-04-09 09:22:22.393950] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully >[2014-04-09 09:22:22.394231] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully >[2014-04-09 09:22:22.394546] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully >[2014-04-09 09:22:22.394824] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully >[2014-04-09 09:22:22.395109] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully >[2014-04-09 09:22:22.395360] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:23.395763] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory >[2014-04-09 09:22:23.396569] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:23.396818] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:22:23.401558] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:22:29.321511] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:22:29.324699] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:29.365199] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:22:29.370017] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:29.410152] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:22:29.416890] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:29.580172] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:29.588998] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:29.596199] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:22:29.596287] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:22:29.612844] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:22:29.771129] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:22:37.207177] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:37.207361] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:37.207474] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:45.721076] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:45.721247] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:45.721304] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:22:45.722916] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:22:45.728314] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket failed (No data available) >[2014-04-09 09:22:45.733896] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1 >[2014-04-09 09:22:45.734000] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:22:45.746974] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes >[2014-04-09 09:22:45.806387] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes >[2014-04-09 09:22:45.806671] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:45.806720] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:45.806823] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory >[2014-04-09 09:22:45.806954] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:45.806989] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:45.807066] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory >[2014-04-09 09:22:45.807128] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:22:45.807184] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory >[2014-04-09 09:22:45.902088] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:46.902738] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied >[2014-04-09 09:22:46.903392] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully >[2014-04-09 09:22:46.903735] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully >[2014-04-09 09:22:46.904051] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully >[2014-04-09 09:22:46.904339] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully >[2014-04-09 09:22:46.904655] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully >[2014-04-09 09:22:46.904935] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully >[2014-04-09 09:22:46.905186] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:47.906655] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory >[2014-04-09 09:22:47.907292] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:22:47.907639] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:22:54.017430] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:22:54.020605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:54.060518] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:22:54.066052] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:54.117782] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:22:54.127677] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:54.264914] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:54.269411] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:22:54.276411] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:22:54.276682] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:22:54.302214] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:22:54.440442] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:23:00.717877] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:00.718083] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:00.718163] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:04.284065] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=on >[2014-04-09 09:23:08.935601] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=on >[2014-04-09 09:23:10.068692] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:10.068839] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:10.068907] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:20.111697] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=on >[2014-04-09 09:23:25.148659] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=on >[2014-04-09 09:23:26.176291] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:26.176413] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:26.176475] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:38.370765] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:38.370887] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:38.370977] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:23:42.044575] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o latency-measurement=off >[2014-04-09 09:23:43.062477] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=hosdu -o count-fop-hits=off >[2014-04-09 09:25:37.980757] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:37.980905] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:37.980964] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:37.982821] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:25:37.989910] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket failed (No data available) >[2014-04-09 09:25:37.995868] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1 >[2014-04-09 09:25:37.996006] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:25:38.000805] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick3 >[2014-04-09 09:25:38.008975] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=hosdu --last=yes >[2014-04-09 09:25:38.068072] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=hosdu --last=yes >[2014-04-09 09:25:38.068345] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:25:38.068398] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:25:38.068539] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/adde8e1c1373801e7292b7a3e1c3ab95.socket error: No such file or directory >[2014-04-09 09:25:38.068669] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_brick_disconnect+0x38) [0x7f239d727158] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:25:38.068704] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:25:38.068763] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/ec0a49e9e8d6b4f6cd581c3f15c8f535.socket error: No such file or directory >[2014-04-09 09:25:38.068804] I [glusterd-utils.c:1207:glusterd_service_stop] 0-management: brick already stopped >[2014-04-09 09:25:38.068854] E [glusterd-utils.c:1518:glusterd_brick_unlink_socket_file] 0-management: Failed to remove /var/run/b66245db8ec106e59be2d08901892e53.socket error: No such file or directory >[2014-04-09 09:25:38.165119] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:25:39.165659] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied >[2014-04-09 09:25:39.166253] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully >[2014-04-09 09:25:39.166582] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully >[2014-04-09 09:25:39.166863] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully >[2014-04-09 09:25:39.167159] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully >[2014-04-09 09:25:39.167432] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully >[2014-04-09 09:25:39.167736] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully >[2014-04-09 09:25:39.167980] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:25:40.168408] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory >[2014-04-09 09:25:40.169232] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13f) [0x31e380f58f] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xcb) [0x31e380df9b] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:25:40.169319] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:25:40.174638] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:25:40.987674] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick1 on port 49152 >[2014-04-09 09:25:40.990819] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:41.030406] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick3 on port 49153 >[2014-04-09 09:25:41.036075] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:41.076219] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /bricks/hosdu_brick5 on port 49154 >[2014-04-09 09:25:41.083389] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:41.220013] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:41.224322] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:41.232363] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:25:41.232923] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:25:41.234948] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:25:41.372776] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=hosdu --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2014-04-09 09:25:46.927453] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:46.928321] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:46.928804] E [glusterd-op-sm.c:6205:glusterd_op_sm] 0-management: lock failed due to Device or resource busy >[2014-04-09 09:25:46.928934] E [glusterd-op-sm.c:6205:glusterd_op_sm] 0-management: lock failed due to Device or resource busy >[2014-04-09 09:25:46.929017] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:48.414344] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:48.414462] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:48.414516] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:49.888768] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:49.888938] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:49.889038] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:51.157743] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:51.157848] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:51.157908] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:52.484470] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:52.484590] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:52.484648] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:53.963238] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:53.963406] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:53.963466] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:55.436341] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:55.436475] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:55.436533] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:59.345817] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:25:59.345956] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 09:25:59.345975] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 09:25:59.345993] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 09:25:59.392605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:25:59.401135] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0 >[2014-04-09 09:25:59.402356] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:59.403469] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:59.422855] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4 >[2014-04-09 09:25:59.428101] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com, port: 0 >[2014-04-09 09:25:59.434819] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 180d8cc5-4883-495f-870a-ea64514eca44 >[2014-04-09 09:25:59.435089] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to rhsauto022.lab.eng.blr.redhat.com (0), ret: 0 >[2014-04-09 09:25:59.461908] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131 >[2014-04-09 09:25:59.461979] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 09:25:59.462212] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:25:59.472978] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 09:25:59.473101] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44 >[2014-04-09 09:25:59.474419] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:01.005950] E [glusterd-brick-ops.c:1232:glusterd_op_stage_add_brick] 0-management: Unable to get replica count >[2014-04-09 09:26:03.283641] I [run.c:190:runner_log] 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=hosdu --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd >[2014-04-09 09:26:03.283806] I [glusterd-brick-ops.c:1023:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it >[2014-04-09 09:26:03.490696] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:26:04.495701] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/67492a140ba2b04dd03d21af9b879f48.socket error: Permission denied >[2014-04-09 09:26:04.496443] I [glusterd-utils.c:4204:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully >[2014-04-09 09:26:04.496744] I [glusterd-utils.c:4209:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully >[2014-04-09 09:26:04.497026] I [glusterd-utils.c:4214:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully >[2014-04-09 09:26:04.497352] I [glusterd-utils.c:4219:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully >[2014-04-09 09:26:04.497634] I [glusterd-utils.c:4224:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully >[2014-04-09 09:26:04.497908] I [glusterd-utils.c:4229:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully >[2014-04-09 09:26:04.504087] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:26:04.505557] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_nodesvc_disconnect+0x4c) [0x7f239d726ccc] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:26:05.506084] E [glusterd-utils.c:4170:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/22ab8e649f7aa4b02ff35208784130ae.socket error: No such file or directory >[2014-04-09 09:26:05.515369] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:26:05.517014] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:26:05.517387] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 09:26:09.620528] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:09.620630] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:14.101851] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:14.101994] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:14.102073] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:15.641234] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:15.641419] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:15.641525] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:18.989750] W [glusterd-rebalance.c:570:glusterd_op_stage_rebalance] 0-management: Missing rebalance-id >[2014-04-09 09:26:29.042577] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 09:26:29.610578] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:26:29.929640] I [glusterd-handshake.c:365:__server_event_notify] 0-: received defrag status updated >[2014-04-09 09:26:29.929726] E [glusterd-rebalance.c:768:glusterd_defrag_event_notify_handle] 0-: Failed to update status >[2014-04-09 09:26:29.936354] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/gluster/gluster-rebalance-95452d6c-d0e6-4934-b9ff-ae485d32048a.sock failed (No data available) >[2014-04-09 09:26:29.948417] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(__glusterd_defrag_notify+0x2d2) [0x7f239d759b22] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f239d7265f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 09:27:36.841904] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:27:36.842181] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:27:36.842313] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:27:37.501911] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:27:37.502107] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 09:27:37.502187] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:46:06.559127] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.40.131:24007 failed (No data available) >[2014-04-09 10:46:06.559620] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held >[2014-04-09 10:46:10.277844] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.36.252:24007 failed (No data available) >[2014-04-09 10:46:10.278418] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held >[2014-04-09 10:46:13.645331] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down >[2014-04-09 10:52:44.685969] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) >[2014-04-09 10:52:44.698572] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory >[2014-04-09 10:52:44.706613] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) >[2014-04-09 10:52:44.706651] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device >[2014-04-09 10:52:44.706668] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed >[2014-04-09 10:52:44.706754] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >[2014-04-09 10:52:50.793857] I [glusterd-store.c:1428:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 4 >[2014-04-09 10:52:51.198271] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0 >[2014-04-09 10:52:51.199914] I [glusterd-handler.c:3320:glusterd_friend_add] 0-management: connect returned 0 >[2014-04-09 10:52:51.200034] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.206776] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.545960] I [glusterd.c:143:glusterd_uuid_init] 0-management: retrieved UUID: 51dc2165-b320-4a27-a602-86580d5e9be5 >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: >+------------------------------------------------------------------------------+ >[2014-04-09 10:52:51.562014] E [socket.c:2162:socket_connect_finish] 0-management: connection to 10.70.36.252:24007 failed (Connection refused) >[2014-04-09 10:52:51.563330] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:52:51.564369] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4 >[2014-04-09 10:52:51.572939] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:52:51.574805] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 10:52:51.575094] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.40.131 (0), ret: 0 >[2014-04-09 10:52:51.647115] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131 >[2014-04-09 10:52:51.647198] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 10:52:51.725609] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.737533] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.749765] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.775698] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:51.796296] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:56.821605] I [rpc-clnt.c:994:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2014-04-09 10:52:56.834913] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 10:52:56.839580] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick1 >[2014-04-09 10:52:56.844625] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 10:52:56.849246] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick3 >[2014-04-09 10:52:56.854591] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 10:52:56.858762] I [glusterd-handler.c:4121:__glusterd_brick_rpc_notify] 0-management: Disconnected from rhsauto057.lab.eng.blr.redhat.com:/bricks/hosdu_brick5 >[2014-04-09 10:52:56.858835] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 10:52:56.858933] I [socket.c:2239:socket_event_handler] 0-transport: disconnecting now >[2014-04-09 10:52:56.859088] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131, port: 0 >[2014-04-09 10:52:56.869072] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 10:52:56.869161] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 10:52:56.869186] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 10:52:56.869201] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 10:52:56.913386] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154 >[2014-04-09 10:52:56.931654] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153 >[2014-04-09 10:52:57.123262] I [glusterd-handshake.c:365:__server_event_notify] 0-: received defrag status updated >[2014-04-09 10:52:57.123312] E [glusterd-rebalance.c:768:glusterd_defrag_event_notify_handle] 0-: Failed to update status >[2014-04-09 10:52:57.135267] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/gluster/gluster-rebalance-95452d6c-d0e6-4934-b9ff-ae485d32048a.sock failed (No data available) >[2014-04-09 10:52:57.144662] E [rpc-transport.c:481:rpc_transport_unref] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(__glusterd_defrag_notify+0x2d2) [0x7f6c5ce51b22] (-->/usr/lib64/glusterfs/3.5qa2/xlator/mgmt/glusterd.so(glusterd_rpc_clnt_unref+0x35) [0x7f6c5ce1e5f5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63) [0x31e380cfe3]))) 0-rpc_transport: invalid argument: this >[2014-04-09 10:52:58.237991] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:01.486679] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:01.492173] I [glusterd-handshake.c:564:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 4 >[2014-04-09 10:53:01.589964] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 10:53:01.590091] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 180d8cc5-4883-495f-870a-ea64514eca44, hostname:rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 10:53:01.590117] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 10:53:01.590330] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 10:53:02.258951] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:06.606095] I [glusterd-handler.c:2458:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 180d8cc5-4883-495f-870a-ea64514eca44 >[2014-04-09 10:53:06.606506] I [glusterd-handler.c:3493:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to rhsauto022.lab.eng.blr.redhat.com (0), ret: 0 >[2014-04-09 10:53:06.616070] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, host: 10.70.40.131 >[2014-04-09 10:53:06.616119] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com >[2014-04-09 10:53:06.616361] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:06.646815] I [glusterd-rpc-ops.c:554:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77 >[2014-04-09 10:53:06.657091] I [glusterd-ping.c:297:glusterd_ping_cbk] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:06.668128] I [glusterd-rpc-ops.c:357:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 180d8cc5-4883-495f-870a-ea64514eca44, host: rhsauto022.lab.eng.blr.redhat.com, port: 0 >[2014-04-09 10:53:06.688215] I [glusterd-handler.c:2620:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 180d8cc5-4883-495f-870a-ea64514eca44 >[2014-04-09 10:53:06.688282] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: 51dc2165-b320-4a27-a602-86580d5e9be5, hostname:rhsauto057.lab.eng.blr.redhat.com >[2014-04-09 10:53:06.688301] I [glusterd-handler.c:2674:__glusterd_handle_friend_update] 0-: Received my uuid as Friend >[2014-04-09 10:53:06.688320] I [glusterd-handler.c:2665:__glusterd_handle_friend_update] 0-: Received uuid: cea2e695-ed21-42bb-854d-28b09fc0bb77, hostname:10.70.40.131 >[2014-04-09 10:53:06.698172] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49152 >[2014-04-09 10:53:06.714191] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154 >[2014-04-09 10:53:06.720195] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153 >[2014-04-09 10:53:12.303712] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 10:53:17.333730] I [glusterd-ping.c:181:glusterd_start_ping] 0-management: defaulting ping-timeout to 10s >[2014-04-09 11:00:09.428152] W [socket.c:522:__socket_rwv] 0-management: readv on 10.70.36.252:24007 failed (No data available) >[2014-04-09 11:00:09.428391] W [glusterd-locks.c:190:glusterd_volume_unlock] 0-: Lock for hosdu not held >[2014-04-09 11:00:09.471205] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49152 >[2014-04-09 11:00:09.484356] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49154 >[2014-04-09 11:00:09.491623] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick (null) on port 49153 >[2014-04-09 11:00:20.416661] E [socket.c:2162:socket_connect_finish] 0-management: connection to 10.70.36.252:24007 failed (Connection refused) >[2014-04-09 11:00:26.181459] W [MSGID: 100032] [glusterfsd.c:1104:cleanup_and_exit] (--> 0-: received signum (15), shutting down >[2014-04-09 11:11:45.133901] I [MSGID: 100030] [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5qa2 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) >[2014-04-09 11:11:45.139978] I [glusterd.c:1161:init] 0-management: Using /var/lib/glusterd as working directory >[2014-04-09 11:11:45.145965] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) >[2014-04-09 11:11:45.146002] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device >[2014-04-09 11:11:45.146018] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed >[2014-04-09 11:11:45.146109] W [rpcsvc.c:1477:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed >[2014-04-09 11:11:48.009226] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 11:11:48.009277] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory) >[2014-04-09 11:11:48.009288] I [glusterd-store.c:1448:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 4 >[2014-04-09 11:11:48.009425] E [store.c:407:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory) >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option transport.socket.listen-backlog 128 > 7: option transport.socket.read-fail-log off > 8: option transport.socket.keepalive-interval 2 > 9: option transport.socket.keepalive-time 10 > 10: option transport-type rdma > 11: option working-directory /var/lib/glusterd > 12: end-volume > 13: >+------------------------------------------------------------------------------+
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1083668
:
884818
| 884819 |
884820
|
889196