Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1486021 Details for
Bug 1624586
Gluster over infiniband causing host timeouts and failures
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
ovirt-12_glusterd.log
ovirt-12_glusterd.log (text/plain), 321.17 KB, created by
housemouse
on 2018-09-22 19:28:18 UTC
(
hide
)
Description:
ovirt-12_glusterd.log
Filename:
MIME Type:
Creator:
housemouse
Created:
2018-09-22 19:28:18 UTC
Size:
321.17 KB
patch
obsolete
>[2018-09-21 21:59:47.704518] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-glusterd: Started running glusterd version 3.12.14 (args: glusterd --xlator-option *.upgrade=on -N) >[2018-09-21 21:59:47.710631] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-21 21:59:47.710687] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-21 21:59:47.710717] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-21 21:59:47.754336] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-21 21:59:47.754380] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-21 21:59:47.754383] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31202 >[2018-09-21 21:59:47.754414] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] >[2018-09-21 21:59:47.760439] I [MSGID: 106194] [glusterd-store.c:3850:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option upgrade on > 9: option event-threads 1 > 10: option ping-timeout 0 > 11: option transport.socket.read-fail-log off > 12: option transport.socket.keepalive-interval 2 > 13: option transport.socket.keepalive-time 10 > 14: option transport-type socket,rdma > 15: option working-directory /var/lib/glusterd > 16: end-volume >[2018-09-21 21:59:47.760881] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f9df2d72e25] -->glusterd(glusterfs_sigwaiter+0xe5) [0x560a8bb234b5] -->glusterd(cleanup_and_exit+0x6b) [0x560a8bb2332b] ) 0-: received signum (15), shutting down > 17: >+------------------------------------------------------------------------------+ >[2018-09-21 21:59:47.761139] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 02:52:34.738556] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 02:52:34.744817] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 02:52:34.744861] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 02:52:34.744873] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 02:52:39.921210] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-22 02:52:39.921278] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-22 02:52:39.921281] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31202 >[2018-09-22 02:52:39.921527] I [MSGID: 106194] [glusterd-store.c:3850:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 02:52:39.922020] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 02:54:10.526357] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f92443c7e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x5606a2ce64b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x5606a2ce632b] ) 0-: received signum (15), shutting down >[2018-09-22 02:54:10.588336] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 02:54:10.594431] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 02:54:10.594470] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 02:54:10.594483] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 02:54:15.816407] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-22 02:54:15.816475] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-22 02:54:15.816478] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31202 >[2018-09-22 02:54:15.816892] I [MSGID: 106194] [glusterd-store.c:3850:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 02:54:15.817570] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 02:54:23.895740] E [MSGID: 101032] [store.c:437:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] >[2018-09-22 02:54:23.895876] I [MSGID: 106477] [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 02:54:27.447183] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 02:54:27.455700] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 02:54:27.459055] I [MSGID: 106129] [glusterd-handler.c:2926:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: ovirt-13.ad.squeakz.net (24007) >[2018-09-22 02:54:27.464153] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 02:54:27.464214] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 02:54:27.464324] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 02:54:27.468952] I [MSGID: 106498] [glusterd-handler.c:3550:glusterd_friend_add] 0-management: connect returned 0 >[2018-09-22 02:54:27.469011] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to ovirt-13.ad.squeakz.net, op_ret: 0, op_errno: 0, ret: 0 >[2018-09-22 02:54:27.469617] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 02:54:27.475724] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 02:54:27.495763] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net >[2018-09-22 02:54:27.495799] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req >[2018-09-22 02:54:27.499886] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 02:54:27.503539] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 02:54:27.508412] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 02:54:27.508462] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 02:54:27.508562] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 02:54:27.513079] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 02:54:27.513108] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 02:54:27.513185] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 02:54:27.525924] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 02:54:27.529485] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 02:54:27.534926] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 02:54:27.538746] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 02:54:27.546128] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 02:54:27.546218] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 02:54:27.549811] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 02:54:27.549935] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 02:54:28.978328] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 02:54:28.986048] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:04:23.744010] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f84e0d7ce25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x5576aef5a4b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x5576aef5a32b] ) 0-: received signum (15), shutting down >[2018-09-22 03:04:23.808727] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 03:04:23.814898] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 03:04:23.814936] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 03:04:23.814949] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 03:04:29.044847] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 03:04:29.046423] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:04:29.046530] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:04:29.046575] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:04:29.046610] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:04:29.046784] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 03:04:29.052188] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:04:29.052219] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:04:29.052299] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 03:04:29.057409] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 03:04:29.057993] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 03:04:33.107788] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 03:04:33.118304] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:04:34.111903] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:04:34.115641] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:04:34.119178] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:04:34.122600] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:04:34.123221] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:04:34.127100] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:04:34.131631] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:04:34.138507] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:04:34.141925] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:04:34.142046] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:04:34.316379] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:04:34.320243] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:04:34.323706] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:04:34.330365] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:04:34.333608] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:04:34.335202] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:16:48.411391] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 03:16:48.411436] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 03:16:48.411512] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 03:17:04.616308] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fe76a293e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55eaf8fbc4b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x55eaf8fbc32b] ) 0-: received signum (15), shutting down >[2018-09-22 03:19:23.611819] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 03:19:23.632029] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 03:19:23.632197] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 03:19:23.632221] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 03:19:29.236887] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 03:19:29.240519] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:19:29.240854] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:19:29.240904] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:19:29.241027] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:19:29.241133] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 03:19:29.246618] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:19:29.246648] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:19:29.246729] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 03:19:29.251692] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 03:19:29.252447] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 03:19:29.253366] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:19:29.257091] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:19:29.272229] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:19:29.275501] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:19:29.282043] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:19:29.285322] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:19:29.291959] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 03:19:29.295316] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:19:29.298700] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:19:29.301930] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:19:29.302065] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:19:29.305070] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:19:29.305180] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:19:29.305236] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:19:29.308460] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:19:29.308585] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:19:29.311810] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:19:29.311931] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:19:29.314008] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:19:29.317621] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:22:51.768164] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.13:24007 failed (Connection timed out) >[2018-09-22 03:22:51.768227] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-13.ad.squeakz.net> (<2eedbeab-db3d-40ad-8d6b-2a7963e647b4>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 03:23:05.720251] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.13:24007 failed (No route to host); disconnecting socket >[2018-09-22 03:25:25.776506] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:25:25.796190] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:25:25.800024] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:25:25.812266] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 03:25:25.819057] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:25:25.822503] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:25:25.826857] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:25:25.830489] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:25:25.832669] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:25:25.838739] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:32:22.908866] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-13.ad.squeakz.net> (<2eedbeab-db3d-40ad-8d6b-2a7963e647b4>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 03:32:28.141232] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:32:28.149450] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:32:33.877551] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:32:33.884453] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:32:33.887591] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:32:33.887771] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:32:33.897094] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 03:32:33.900729] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:32:33.904342] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:32:33.907037] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:33:02.555991] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f7f6d7c7e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x560e06f3f4b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x560e06f3f32b] ) 0-: received signum (15), shutting down >[2018-09-22 03:33:02.618264] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 03:33:02.623711] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 03:33:02.623747] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 03:33:02.623759] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 03:33:07.813812] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 03:33:07.815445] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:33:07.815551] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 03:33:07.815596] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:33:07.815631] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:33:07.815723] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 03:33:07.820747] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 03:33:07.820784] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 03:33:07.820877] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 03:33:07.825444] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 03:33:07.826246] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 03:33:11.041351] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 03:33:11.052248] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:33:12.633451] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:33:12.638131] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:33:12.641443] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:33:12.644405] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:33:12.644848] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:33:12.648388] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:33:12.652870] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:33:12.659850] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:33:12.663337] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:33:12.663465] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:33:12.983647] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:33:12.987545] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:33:12.990896] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:33:12.998657] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:33:13.002724] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:33:13.003366] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 03:33:22.159226] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req >[2018-09-22 03:40:04.652024] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (No data available) >[2018-09-22 03:40:04.652104] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 03:40:09.991197] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:40:09.999250] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:40:14.683575] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:40:14.691665] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:40:14.694964] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:40:14.696782] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:40:14.705675] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:40:14.709375] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:40:14.712463] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:40:14.716689] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:40:55.356358] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 03:41:06.744402] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 03:43:16.287043] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:43:16.298301] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:43:16.302302] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:43:16.310083] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:43:16.313538] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:43:16.316806] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:43:16.329670] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:43:16.333094] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:43:16.337082] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:43:16.347003] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:46:18.851217] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 03:46:24.136766] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 03:46:24.144682] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:46:28.884597] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 03:46:28.892733] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:46:28.896337] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:46:28.896519] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:46:28.905699] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 03:46:28.909309] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 03:46:28.912549] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 03:46:28.916357] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:13:23.980836] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:13:23.989323] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:13:23.989571] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to ceph3.squeakz.lab, op_ret: 0, op_errno: 0, ret: 0 >[2018-09-22 04:13:23.994555] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:13:23.998199] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:13:24.342610] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:13:24.679225] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:13:24.686699] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:13:24.335159] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:16:48.742093] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 04:16:48.742162] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 04:16:48.742435] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 04:21:54.366971] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh --volname=gluster_vmstore >[2018-09-22 04:21:54.953095] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 04:21:54.996996] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o nfs.disable=on --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:55.430516] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o user.cifs=disable --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:55.882271] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o auth.allow=* --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:58.070306] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o performance.quick-read=off performance.read-ahead=off performance.io-cache=off performance.low-prio-threads=32 network.remote-dio=enable cluster.eager-lock=enable cluster.quorum-type=auto cluster.server-quorum-type=server cluster.data-self-heal-algorithm=full cluster.locking-scheme=granular cluster.shd-max-threads=8 cluster.shd-wait-qlength=10000 features.shard=on user.cifs=off --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:58.528449] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o storage.owner-uid=36 --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:58.961879] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o storage.owner-gid=36 --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:59.509785] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o server.allow-insecure=on --gd-workdir=/var/lib/glusterd >[2018-09-22 04:21:59.960744] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7fef9f1bb49a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7fef9f1baf5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fefaa6bbdd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o network.ping-timeout=30 --gd-workdir=/var/lib/glusterd >[2018-09-22 04:22:43.255709] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 04:22:43.257490] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 04:23:23.276528] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 04:23:24.114346] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 04:23:24.115050] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:23:24.140417] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 04:23:24.140560] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:23:24.140586] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:23:24.140902] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 04:23:24.141878] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:23:24.141907] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:23:24.141960] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:23:25.172393] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 04:23:25.172767] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 04:23:25.172963] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:23:25.172995] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:23:25.173041] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 04:23:25.173208] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:23:25.173234] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:27:19.416657] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:27:19.417179] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fef9f0ff1ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fef9f109128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fef9f1c0cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:27:19.417225] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:27:19.417260] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gluster_vmstore. Starting local bricks. >[2018-09-22 04:27:19.442453] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fefaa66eebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fefaa433d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fefaa433ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fefaa435640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fefaa436130] ))))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2018-09-22 04:23:43.568645 (xid=0x56) >[2018-09-22 04:27:19.442492] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on ovirt-11.ad.squeakz.net. Please check log file for details. >[2018-09-22 04:27:19.442674] I [socket.c:3679:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1) >[2018-09-22 04:27:19.442688] E [rpcsvc.c:1364:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management) >[2018-09-22 04:27:19.442710] E [MSGID: 106430] [glusterd-utils.c:568:glusterd_submit_reply] 0-glusterd: Reply submission failed >[2018-09-22 04:27:23.085894] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:27:23.090066] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:27:30.227514] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:27:30.234792] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:27:30.234835] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:27:30.236592] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 11310 >[2018-09-22 04:27:30.236737] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:27:30.239776] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:27:30.240000] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:27:30.249213] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 04:27:30.252604] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:27:30.255764] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:27:30.258999] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:27:31.237039] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:27:35.238116] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:27:35.238302] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:27:35.238356] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:27:35.238432] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:27:35.238460] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:27:35.238531] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:27:35.238551] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:27:43.221606] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:27:50.223148] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:27:57.224471] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:04.225801] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:11.227114] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:18.228389] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:25.229684] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:32.231089] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:39.232539] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:46.233935] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:28:53.235501] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:00.236826] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:07.238439] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:14.240010] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:21.241396] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:28.242718] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:35.244087] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:42.245458] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:49.246760] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:29:56.248036] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:03.249609] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:10.251197] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:17.252649] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:24.254038] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:31.255533] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:38.257064] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:45.258557] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:52.259936] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:30:59.261374] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:06.262589] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:13.264041] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:20.265435] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:27.267236] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:34.268778] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:41.270281] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:48.271629] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:31:55.273026] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:02.274639] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:09.276130] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:16.277640] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:23.279041] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:30.280600] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:37.281992] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:43.820632] I [MSGID: 106542] [glusterd-utils.c:8228:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 11283 >[2018-09-22 04:32:44.283439] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:32:44.846489] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:32:44.846530] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:32:44.847017] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:32:44.847056] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:32:44.847184] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:32:44.847204] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:32:44.847274] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:32:44.847292] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:32:44.847445] I [MSGID: 106144] [glusterd-pmap.c:396:pmap_registry_remove] 0-pmap: removing brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 04:33:40.238049] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 04:33:40.341468] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 04:33:40.342150] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:33:40.367953] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:33:40.367991] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:33:40.369269] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:33:40.369306] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:33:40.369374] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:33:41.401905] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:33:41.401995] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:33:41.402073] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:33:41.402111] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:34:12.810205] W [socket.c:600:__socket_rwv] 0-management: readv on /var/run/gluster/09a0d99ac750f199f4d0b8e5b92ecfc3.socket failed (No data available) >[2018-09-22 04:34:12.817935] W [glusterd-handler.c:6064:__glusterd_brick_rpc_notify] 0-management: got disconnect from stale rpc on /gluster-bricks/brick1/brick1 >[2018-09-22 04:34:13.679389] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 04:34:26.786331] I [MSGID: 106533] [glusterd-volume-ops.c:920:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gluster_vmstore >[2018-09-22 04:36:20.519279] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:36:20.519759] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fef9f0ff1ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fef9f109128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fef9f1c0cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:36:20.519779] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:37:31.545243] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:37:31.545758] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fef9f0ff1ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fef9f109128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fef9f1c0cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:37:31.545789] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:37:31.546410] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fefaa66eebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fefaa433d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fefaa433ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fefaa435640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fefaa436130] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2018-09-22 04:36:31.321300 (xid=0x67) >[2018-09-22 04:37:31.546447] E [MSGID: 106167] [glusterd-handshake.c:2193:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later >[2018-09-22 04:37:31.546760] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fefaa66eebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fefaa433d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fefaa433ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fefaa435640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fefaa436130] ))))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2018-09-22 04:36:46.883317 (xid=0x68) >[2018-09-22 04:37:31.546799] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on ovirt-11.ad.squeakz.net. Please check log file for details. >[2018-09-22 04:37:37.051172] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:37:37.059978] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:37:42.343117] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:37:42.351067] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:37:42.351114] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:37:42.352894] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 12363 >[2018-09-22 04:37:42.727896] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:37:42.735752] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:37:42.739937] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:37:42.761146] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 04:37:42.764601] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:37:42.767968] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:37:42.771383] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:37:43.353363] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:37:47.354349] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:37:47.354564] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:37:47.354604] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:37:47.354683] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:37:47.354706] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:37:47.354781] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:37:47.354803] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:37:55.337955] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:02.339422] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:09.340737] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:13.003510] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 04:38:16.342390] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:23.343874] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:30.345634] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:37.347024] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:44.348712] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:51.350140] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:58.351560] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:05.352973] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:38:13.324797] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 04:39:12.354482] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:19.355937] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:26.357261] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:33.358621] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:40.359972] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:47.361443] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:39:54.363009] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:40:01.364568] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:40:08.366036] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:40:10.005367] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:40:10.005856] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fef9f0ff1ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fef9f109128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fef9f1c0cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:40:10.005881] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:40:10.006647] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fefaa66eebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fefaa433d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fefaa433ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fefaa435640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fefaa436130] ))))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2018-09-22 04:39:32.195097 (xid=0x77) >[2018-09-22 04:40:10.006694] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on ovirt-11.ad.squeakz.net. Please check log file for details. >[2018-09-22 04:40:15.367676] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:40:15.489090] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:40:15.500086] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:40:22.369313] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 04:40:23.382289] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:40:23.389442] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:40:23.389490] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:40:23.390967] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:40:23.391012] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:40:23.391091] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:40:24.423518] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:40:24.423601] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:40:24.423681] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:40:24.423715] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:40:24.423797] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:40:24.423828] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:40:24.448697] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:40:24.452229] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:40:24.452686] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:40:28.437034] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 04:40:28.440511] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:40:28.443729] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:40:28.445251] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:46:55.739423] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 04:46:55.749158] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 04:46:55.749205] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 04:46:55.749219] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 04:47:01.383243] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 04:47:01.385796] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 04:47:01.613949] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 04:47:01.614364] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 04:47:01.614428] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 04:47:01.614464] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:47:01.614581] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 04:47:01.619896] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 04:47:01.619930] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:47:01.620010] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 04:47:01.625309] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 04:47:01.626628] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:47:01.629737] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:47:01.640301] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:47:01.644093] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:47:01.658182] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gluster_vmstore. Starting local bricks. >[2018-09-22 04:47:01.659029] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 04:47:01.661899] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:47:01.687043] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 04:47:01.687308] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:47:01.687356] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:47:01.688161] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 04:47:01.689687] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:47:01.689728] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:47:01.689777] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:47:02.693491] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 04:47:02.693832] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:47:02.693869] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:47:02.693913] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 04:47:02.694090] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:47:02.694114] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:47:02.694154] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 04:47:02.694321] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:47:02.694344] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:47:02.715973] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 04:47:02.720172] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:47:02.723789] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:47:02.730533] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 04:47:02.734094] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:47:02.737254] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:47:02.745404] I [socket.c:2481:socket_event_handler] 0-transport: EPOLLERR - disconnecting now >[2018-09-22 04:47:02.753427] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 04:47:02.756838] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:47:02.759984] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:47:02.760155] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:47:02.760197] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 04:47:02.763785] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:47:02.764046] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:47:02.767404] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:47:02.767584] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:47:02.767623] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:47:02.768898] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 3318 >[2018-09-22 04:47:02.771019] W [socket.c:600:__socket_rwv] 0-glustershd: readv on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket failed (No data available) >[2018-09-22 04:47:02.771056] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 04:47:03.769222] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:47:03.769318] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:47:04.772329] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:47:04.772383] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:47:04.772459] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:47:04.772484] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:47:04.772569] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:47:04.772595] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:47:04.798799] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:48:13.826327] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (Connection timed out) >[2018-09-22 04:47:04.798906] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:48:13.826375] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:48:13.826514] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f32e16a11ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f32e16ab128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f32e1762cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:48:13.826532] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:58:08.686471] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 04:58:08.699146] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 04:58:08.699219] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 04:58:08.699245] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 04:58:14.422736] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 04:58:14.423912] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 04:58:14.644104] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 04:58:14.644214] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 04:58:14.644273] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 04:58:14.644315] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:58:14.644415] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 04:58:19.656310] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 04:58:19.656363] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:58:19.656454] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 04:58:24.671419] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 04:58:55.666717] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.13:24007 failed (Connection reset by peer) >[2018-09-22 04:58:55.666808] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-13.ad.squeakz.net> (<2eedbeab-db3d-40ad-8d6b-2a7963e647b4>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:58:55.667508] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fce467e51ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fce467ef128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fce468a6cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:58:55.667533] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:58:55.667567] C [MSGID: 106002] [glusterd-server-quorum.c:360:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume gluster_vmstore. Stopping local bricks. >[2018-09-22 04:58:55.668755] E [MSGID: 106029] [glusterd-utils.c:8188:glusterd_brick_signal] 0-glusterd: Unable to open pidfile: /var/run/gluster/vols/gluster_vmstore/10.0.14.12-gluster-bricks-brick1-brick1.pid [No such file or directory] >[2018-09-22 04:58:55.669700] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fce51d54ebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fce51b19d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fce51b19ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fce51b1b640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fce51b1c130] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2018-09-22 04:58:24.671758 (xid=0x2) >[2018-09-22 04:58:55.669742] E [MSGID: 106167] [glusterd-handshake.c:2193:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later >[2018-09-22 04:59:05.702605] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.13:24007 failed (Connection refused); disconnecting socket >[2018-09-22 04:59:06.390735] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 04:59:06.398928] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:59:06.403387] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 04:59:06.410063] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gluster_vmstore. Starting local bricks. >[2018-09-22 04:59:06.411010] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 04:59:06.415324] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 04:59:06.436732] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 04:59:06.436973] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 04:59:06.437026] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 04:59:06.437630] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 04:59:06.438947] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 04:59:06.438982] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 04:59:06.439016] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 04:59:07.442798] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 04:59:07.443125] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 04:59:07.443167] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 04:59:07.443214] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 04:59:07.443397] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 04:59:07.443421] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 04:59:07.443462] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 04:59:07.443631] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 04:59:07.443654] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 04:59:07.469824] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 04:59:07.470374] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:59:07.473687] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:59:07.484507] I [socket.c:2481:socket_event_handler] 0-transport: EPOLLERR - disconnecting now >[2018-09-22 04:59:07.492784] I [MSGID: 106005] [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management: Brick 10.0.14.12:/gluster-bricks/brick1/brick1 has disconnected from glusterd. >[2018-09-22 04:59:07.546440] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 04:59:08.492069] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 04:59:08.495270] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:59:08.498518] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 04:59:08.500060] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 04:59:46.556002] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (Connection timed out) >[2018-09-22 04:59:46.556095] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 04:59:46.556269] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7fce467e51ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7fce467ef128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7fce468a6cf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 04:59:46.556288] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 04:59:46.556310] C [MSGID: 106002] [glusterd-server-quorum.c:360:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume gluster_vmstore. Stopping local bricks. >[2018-09-22 04:59:46.557468] I [MSGID: 106542] [glusterd-utils.c:8228:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 16191 >[2018-09-22 04:59:47.557791] I [MSGID: 106144] [glusterd-pmap.c:396:pmap_registry_remove] 0-pmap: removing brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 05:00:00.465156] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fce50bb2e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55ed06d5f4b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x55ed06d5f32b] ) 0-: received signum (15), shutting down >[2018-09-22 05:03:08.741979] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 05:03:08.750031] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 05:03:08.750085] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 05:03:08.750100] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 05:03:14.504129] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 05:03:14.505220] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 05:03:14.743510] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 05:03:14.743631] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 05:03:14.743691] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 05:03:14.743749] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 05:03:14.743847] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 05:03:19.756068] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 05:03:19.756117] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 05:03:19.756220] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 05:03:24.763109] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 05:03:24.764279] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:03:24.777987] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 05:03:29.789814] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:03:29.806979] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gluster_vmstore. Starting local bricks. >[2018-09-22 05:03:29.808273] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 05:03:29.812816] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 05:03:29.842029] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (No route to host); disconnecting socket >[2018-09-22 05:03:29.842079] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:03:29.842426] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 05:03:29.845635] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 05:03:29.845879] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:03:29.845917] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:03:29.846546] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 05:03:29.848023] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 05:03:29.848060] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:03:29.848110] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 05:03:30.852268] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 05:03:30.852554] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:03:30.852588] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:03:30.852632] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 05:03:30.852810] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:03:30.852834] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:03:30.852874] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 05:03:30.853041] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:03:30.853064] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:03:30.879459] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 05:03:30.879928] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 05:03:30.883364] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:04:00.927017] I [socket.c:2481:socket_event_handler] 0-transport: EPOLLERR - disconnecting now >[2018-09-22 05:04:30.968434] I [MSGID: 106005] [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management: Brick 10.0.14.12:/gluster-bricks/brick1/brick1 has disconnected from glusterd. >[2018-09-22 05:04:30.970364] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 05:04:30.978247] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:04:30.978382] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 05:04:31.109341] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 05:04:35.062521] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 05:07:36.003610] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:08:03.498956] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:08:03.516110] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:08:08.529229] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:08:08.537679] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:08:08.537729] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:08:08.539315] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 3537 >[2018-09-22 05:08:08.552311] W [socket.c:600:__socket_rwv] 0-glustershd: readv on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket failed (No data available) >[2018-09-22 05:08:08.552377] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 05:08:08.730208] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:08:08.734333] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:08:08.737978] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:08:09.540170] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:08:13.540743] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:13.540866] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:08:13.540892] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:08:13.540964] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:08:13.540988] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:08:13.541057] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:08:13.541077] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:08:13.565643] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:08:13.569105] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:08:13.569208] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:08:20.949245] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:27.950999] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:34.954108] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:41.955317] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:48.956760] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:08:55.959384] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:02.960579] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:09.962050] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:15.002581] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:09:16.964333] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:23.966751] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:30.968105] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:37.969628] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:44.970910] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:51.972210] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:09:58.973478] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:05.975341] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:12.977745] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:19.978810] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:26.980119] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:33.981606] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:40.983981] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:47.985464] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:10:54.986827] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:01.988392] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:08.989725] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:15.991758] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:22.992816] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:29.994178] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:36.995338] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:39.231795] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (No data available) >[2018-09-22 05:11:39.231871] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:11:39.232451] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 05:11:39.232476] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 05:11:43.996520] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:50.997924] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:11:57.004936] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 05:12:02.005496] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:12:13.016662] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:12:15.971048] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:12:24.028248] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:12:35.040560] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:12:46.056814] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:12:57.068864] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:13:08.077737] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:13:16.286998] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:13:19.089176] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:13:30.103196] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:13:41.113020] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:13:52.124072] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:03.134097] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:10.135430] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:17.136663] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:28.148236] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:39.156017] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:46.163417] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:53.171820] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:00.178588] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:07.186470] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:14.195318] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:14:16.609146] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:15:21.202865] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:28.209756] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:32.971391] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:15:32.982645] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:15:32.987501] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:15:32.995706] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:15:32.995761] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:15:32.997615] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 05:15:35.216123] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:15:35.216284] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:15:35.216586] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 05:15:35.253346] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:15:35.253458] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:15:35.253548] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:15:35.253576] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:15:35.253650] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:15:35.253674] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:15:35.277621] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:15:35.281675] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:15:35.284902] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:15:35.285026] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:15:35.285233] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:15:35.288416] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:15:36.109885] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:16:17.243383] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:16:49.066326] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 05:16:49.066388] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 05:16:49.066584] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 05:18:17.873687] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:18:25.695742] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:18:25.696426] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 05:18:25.696452] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 05:18:37.246742] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 05:18:17.873687] and [2018-09-22 05:18:47.732837] >[2018-09-22 05:19:23.024631] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:19:23.032701] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:19:23.036932] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:19:23.045582] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:19:23.045657] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:19:23.049110] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 6786 >[2018-09-22 05:19:23.052874] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:19:23.055986] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:19:23.061270] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 05:19:23.081218] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:19:23.096741] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:19:23.100226] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:19:23.103420] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:19:23.118844] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:19:24.049737] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:19:28.050676] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:19:28.050874] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:19:28.050909] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:19:28.050987] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:19:28.051007] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:19:28.051095] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:19:28.051133] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:19:35.328935] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:19:42.330519] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:19:49.331888] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:19:56.333351] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:03.335527] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:10.336992] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:17.339288] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:24.340514] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:31.341706] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:38.343061] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:45.345039] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:52.347413] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:20:59.348812] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:06.350561] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:13.351718] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:20.353956] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:27.355105] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:34.356465] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:41.357876] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:48.359267] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:21:55.360887] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:02.362937] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:09.364482] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:16.365654] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:19.140974] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:22:23.367054] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:30.368340] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:37.369466] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:44.370653] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:51.371920] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:22:58.372998] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:05.374232] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:12.375359] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:19.376702] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:26.378075] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:33.379332] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:40.380966] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:47.382844] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:23:54.384803] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:01.386628] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:08.388335] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:15.389633] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:19.820406] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:24:22.390790] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:29.392314] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:36.393593] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:43.395201] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:50.396585] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:24:57.397939] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:04.400056] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:11.401716] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:18.402920] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:25.404555] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:32.406779] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:39.408189] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:46.409455] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:25:53.410821] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:00.412495] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:07.414212] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:14.415436] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:21.416749] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:28.418158] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:35.420393] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:42.421671] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:49.423163] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:26:56.424639] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:03.425927] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:10.427220] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:17.428525] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:20.786598] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:27:24.430294] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:31.431451] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:38.433233] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:45.434598] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:52.435969] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:27:59.437199] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:06.438695] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:13.439908] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:20.441169] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:27.442518] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:34.443876] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:41.445206] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:46.071546] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:28:46.071585] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:28:46.071713] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:28:46.071732] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:28:46.071854] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:28:46.071873] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:28:46.108202] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gluster_vmstore -o performance.quick-read=off performance.read-ahead=off performance.io-cache=off performance.low-prio-threads=32 network.remote-dio=enable cluster.eager-lock=enable cluster.quorum-type=auto cluster.server-quorum-type=server cluster.data-self-heal-algorithm=full cluster.locking-scheme=granular cluster.shd-max-threads=8 cluster.shd-wait-qlength=10000 features.shard=on user.cifs=off --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:46.136110] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o performance.quick-read=off performance.read-ahead=off performance.io-cache=off performance.low-prio-threads=32 network.remote-dio=enable cluster.eager-lock=enable cluster.quorum-type=auto cluster.server-quorum-type=server cluster.data-self-heal-algorithm=full cluster.locking-scheme=granular cluster.shd-max-threads=8 cluster.shd-wait-qlength=10000 features.shard=on user.cifs=off --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:46.627856] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:28:46.627867] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:28:46.628017] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:28:46.655101] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gluster_vmstore -o storage.owner-uid=36 --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:46.678229] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o storage.owner-uid=36 --gd-workdir=/var/lib/glusterd >The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped" repeated 2 times between [2018-09-22 05:28:46.071585] and [2018-09-22 05:28:47.109548] >[2018-09-22 05:28:47.109661] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:28:47.109721] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:28:47.142404] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gluster_vmstore -o storage.owner-gid=36 --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:47.169698] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o storage.owner-gid=36 --gd-workdir=/var/lib/glusterd >The message "I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped" repeated 3 times between [2018-09-22 05:28:46.071546] and [2018-09-22 05:28:47.637161] >[2018-09-22 05:28:47.637172] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:28:47.637384] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:28:47.663735] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gluster_vmstore -o server.allow-insecure=on --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:47.686695] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f95a702549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f95a7024f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f95b2525dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gluster_vmstore -o server.allow-insecure=on --gd-workdir=/var/lib/glusterd >[2018-09-22 05:28:48.447482] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:55.449916] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:02.451132] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:09.452282] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:16.453535] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:28:21.121450] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:28:47.637427] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >The message "I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped" repeated 2 times between [2018-09-22 05:28:46.628017] and [2018-09-22 05:28:47.637541] >The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped" repeated 3 times between [2018-09-22 05:28:46.071873] and [2018-09-22 05:28:47.637549] >[2018-09-22 05:29:23.454937] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:30.456204] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:37.457490] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:44.458909] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:51.460219] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:29:58.461530] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:05.464202] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:12.465405] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:19.466850] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:21.769762] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:30:26.468152] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:33.469314] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:40.471197] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:47.472351] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:30:54.473526] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:01.474778] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:08.476053] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:15.477797] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:22.479133] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:22.131137] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:31:29.481112] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:36.482533] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:43.483768] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:50.485102] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:31:57.487044] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:04.489476] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:11.491483] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:18.493154] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:22.528104] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:32:25.494305] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:32.495648] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:39.496717] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:46.497924] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:32:53.499114] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:00.500341] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:07.502262] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:14.503431] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:21.504688] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:28.505998] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:35.507282] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:42.508710] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:49.510729] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:33:56.512841] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:03.514922] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:10.516103] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:17.517489] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:24.518693] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:31.519900] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:38.521212] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:45.522483] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:52.523652] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:34:59.524864] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:06.526917] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:13.528099] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:20.530133] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:23.537954] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:35:27.531375] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:34.532553] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:41.533714] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:48.534997] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:35:50.511445] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:35:50.512214] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 05:35:50.512241] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 05:35:55.536168] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:02.538176] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:04.546979] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 05:36:09.547505] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:16.558091] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:23.568351] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:30.574756] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:37.582288] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:44.591390] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:48.291776] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:36:48.296539] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:36:48.296601] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:36:48.298794] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 05:36:51.604167] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:36:51.604342] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:36:51.604517] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 05:36:51.661950] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:36:51.662073] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:36:51.662197] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:36:51.662242] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:36:51.662384] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:36:51.662426] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:36:51.688526] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:36:51.691186] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:36:51.698123] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:36:51.703778] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:36:51.710561] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:36:51.714047] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:36:51.714271] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:35:29.923087] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:39:24.905354] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:41:25.569732] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:42:12.656224] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:42:12.656717] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 05:42:12.656742] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 05:42:24.658991] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 05:43:26.223460] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:44:26.547107] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:45:31.541494] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:45:31.638085] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:45:31.642066] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:45:31.648681] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:45:31.652211] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:45:31.652325] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:45:31.655751] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 8094 >[2018-09-22 05:45:31.655826] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:45:31.659510] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:45:31.668820] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 05:45:32.656096] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:45:36.656970] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:45:36.657170] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:45:36.657211] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:45:36.657318] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:45:36.657373] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:45:36.657454] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:45:36.657477] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:45:36.684690] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:45:36.687843] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:45:36.688079] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:45:43.984832] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:45:50.987056] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:45:57.988226] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:04.989540] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:11.990814] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:18.992781] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:25.994210] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:32.995354] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:39.996826] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:46.998101] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:46:54.000472] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:01.002048] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:08.003957] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:15.005411] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:22.007088] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:45:36.688967] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:47:29.008603] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:36.009991] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:37.336247] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:47:43.011473] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:50.012615] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:47:57.014014] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:04.015547] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:11.017542] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:18.019026] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:25.020374] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:32.021816] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:39.023287] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:46.024635] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:48:53.025902] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:00.027156] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:07.028484] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:14.029782] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:21.031954] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:26.738480] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:49:28.033900] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:35.035198] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:42.037537] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:49.038811] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:49:56.040152] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:03.042363] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:10.043792] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:17.045779] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:24.047182] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:31.048449] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:38.049697] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:45.051427] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:52.052759] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:50:59.054101] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:06.055275] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:13.056650] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:20.057943] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:27.059102] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 05:49:26.738480] and [2018-09-22 05:49:37.979984] >[2018-09-22 05:51:34.060622] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:41.061854] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:48.064345] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:51:55.065575] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:02.067905] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:09.069404] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:16.070740] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:23.072030] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:30.073156] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:37.074368] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:38.934637] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:52:44.075701] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:51.077029] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:52:58.078351] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:05.079509] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:12.082049] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:19.083340] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:26.084609] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:33.086027] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:35.717249] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 05:53:35.717951] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 05:53:35.717978] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 05:53:39.252247] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:53:40.087131] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:47.089241] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:53:49.098681] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 05:53:54.097070] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:01.104901] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:08.111042] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:15.117039] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:22.123172] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:29.129702] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:33.110655] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 05:54:33.127077] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:54:33.131326] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 05:54:33.138150] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 05:54:33.138206] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 05:54:33.140081] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 05:54:36.136712] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 05:54:36.136876] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 05:54:36.137045] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 05:54:36.172609] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 05:54:36.172701] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 05:54:36.172788] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 05:54:36.172822] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 05:54:36.172896] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 05:54:36.172923] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 05:54:36.201627] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:54:36.204674] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:54:36.204823] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:54:38.220799] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 05:54:38.228824] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:54:38.231969] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 05:54:38.234281] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 05:54:43.245971] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:56:43.889852] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 05:59:44.834314] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:01:58.219483] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 06:01:58.219483] and [2018-09-22 06:02:45.790104] >[2018-09-22 06:05:46.709434] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:07:47.326602] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 06:07:47.326602] and [2018-09-22 06:08:19.977920] >[2018-09-22 06:09:47.988493] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:11:48.648023] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:14:49.592668] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:15:49.935155] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:16:49.380564] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 06:16:49.380618] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 06:16:49.381205] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 06:18:50.850964] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:20:48.377335] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 06:20:48.377335] and [2018-09-22 06:20:51.480452] >[2018-09-22 06:27:04.826934] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:27:05.138427] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:27:53.664029] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:31:54.920336] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:32:55.232832] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:37:56.819412] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:39:30.375773] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:39:30.694869] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:41:58.084910] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:42:58.395011] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:44:59.009405] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:51:58.365207] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 3 times between [2018-09-22 06:51:58.365207] and [2018-09-22 06:53:01.496949] >[2018-09-22 06:54:01.806550] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:56:02.438736] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:58:21.594128] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 06:58:21.934349] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:00:03.672024] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:07:05.828505] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:08:06.138551] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:09:06.469517] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:10:34.114642] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:10:34.432427] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:14:08.037978] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:16:48.993743] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:16:49.687957] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 07:16:49.688012] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 07:16:49.688222] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 07:16:49.318214] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:22:10.521019] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 07:22:10.521019] and [2018-09-22 07:23:05.327380] >[2018-09-22 07:25:11.509797] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:31:13.359373] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:32:13.704592] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:34:14.332245] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:38:15.573052] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:48:18.678669] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:49:18.987979] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:50:19.328733] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:52:19.963152] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:59:16.799924] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 07:59:17.133773] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:00:22.470231] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:01:22.789611] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:05:32.641690] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 08:05:32.641690] and [2018-09-22 08:06:24.372191] >[2018-09-22 08:11:47.193885] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:11:47.514706] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:14:26.898743] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:15:27.221481] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:16:27.535510] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:18:01.501006] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:18:01.819960] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:21:29.095638] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:23:29.724508] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:28:31.275823] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:32:32.540356] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:36:24.533357] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:36:24.857714] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:37:34.104363] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:41:35.336977] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 08:41:35.336977] and [2018-09-22 08:42:40.613936] >[2018-09-22 08:44:36.277430] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:46:36.906628] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:47:37.222924] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:49:37.887794] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:52:38.830973] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:54:53.793124] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:54:54.117687] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:55:39.764345] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 08:58:40.690567] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:06:43.173330] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:09:44.117587] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:10:44.430006] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:12:45.039166] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:16:50.301975] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 09:16:50.302040] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 09:16:50.302564] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 09:18:46.929469] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 09:18:46.929469] and [2018-09-22 09:18:57.898169] >[2018-09-22 09:23:48.481002] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:37:03.876095] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:37:04.186011] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:43:54.649167] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:49:56.526326] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:55:20.606170] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:55:20.931888] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 09:57:58.998107] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:00:59.924664] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:02:00.253208] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:03:00.562443] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:05:01.212106] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:12:03.431155] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:13:03.743572] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:19:05.652860] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:19:34.364562] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 10:19:34.364562] and [2018-09-22 10:21:06.280397] >[2018-09-22 10:27:08.132433] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:28:08.446718] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:29:08.766445] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:31:56.757279] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 10:31:56.757279] and [2018-09-22 10:32:09.697096] >[2018-09-22 10:34:10.326468] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:35:10.645915] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:36:10.975507] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:37:11.302396] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:38:11.622288] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:39:11.957416] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:40:12.276938] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:43:13.209431] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:46:14.131020] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:50:15.400149] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:52:16.025630] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:53:16.333429] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:55:16.957016] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:57:17.586739] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 10:58:17.897419] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:01:18.848531] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:14:22.885383] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:19:24.422384] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:20:05.919855] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 11:20:05.919855] and [2018-09-22 11:20:24.748676] >[2018-09-22 11:23:25.690597] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:24:26.018483] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:27:26.935437] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:31:28.193204] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:37:30.073768] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:42:31.633639] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:44:32.241475] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:45:32.565440] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:46:32.886492] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:48:33.502368] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:49:33.817568] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:50:34.146399] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:51:34.473854] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:54:35.430316] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:55:35.756915] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:57:36.401594] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 11:59:37.008196] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:01:37.644879] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:05:38.911606] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:09:40.165373] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:14:41.786552] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:23:44.566138] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:27:45.825622] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:28:46.134972] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:31:47.083759] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 12:31:47.083759] and [2018-09-22 12:33:10.482983] >[2018-09-22 12:34:48.018613] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:36:48.642113] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:37:48.965887] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:40:49.907759] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:43:50.849739] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:51:53.314545] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:52:53.631572] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:54:54.255437] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:55:54.578627] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 12:57:55.212177] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:01:56.455433] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:09:16.598584] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:09:16.933165] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:09:58.928499] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:21:35.209525] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 13:21:35.209525] and [2018-09-22 13:23:03.022442] >[2018-09-22 13:28:04.553435] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:35:52.927586] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 13:35:52.928002] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f95a6f691ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f95a6f73128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f95a702acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 13:35:52.928026] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 13:36:04.390695] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 13:37:12.072784] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 13:37:12.083891] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:37:12.087221] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 13:37:12.095248] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:37:12.095317] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:37:12.097672] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 9075 >[2018-09-22 13:37:12.097759] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 13:37:12.100862] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:37:12.103856] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:37:12.112795] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 13:37:12.121999] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:37:12.125208] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:37:12.131772] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:37:12.141654] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:37:13.098042] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:37:17.098866] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:17.099103] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 13:37:17.099153] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 13:37:17.099277] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:37:17.099323] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:37:17.099421] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:37:17.099454] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:37:24.493887] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:31.494986] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:38.496192] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:45.497217] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:52.498269] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:37:59.499358] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:38:06.500464] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:38:13.501453] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:38:14.023011] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f95b1336e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x561fd4c184b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x561fd4c1832b] ) 0-: received signum (15), shutting down >[2018-09-22 13:39:05.425302] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 13:39:05.432599] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 13:39:05.432653] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 13:39:05.432669] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 13:39:10.757354] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 13:39:10.758440] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 13:39:10.992289] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 13:39:10.992415] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 13:39:10.992472] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 13:39:10.992530] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:39:10.992644] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 13:39:10.999235] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 13:39:10.999296] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:39:10.999389] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 13:39:11.004318] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 13:39:11.182714] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 13:39:11.186567] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gluster_vmstore. Starting local bricks. >[2018-09-22 13:39:11.187636] I [glusterd-utils.c:5962:glusterd_brick_start] 0-management: discovered already-running brick /gluster-bricks/brick1/brick1 >[2018-09-22 13:39:11.187716] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:39:11.209808] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:39:11.212844] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:39:11.219766] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 13:39:11.219939] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:39:11.219978] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:39:11.220386] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 13:39:11.221472] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 13:39:11.221508] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:39:11.221570] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 13:39:12.224794] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 13:39:12.225099] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 13:39:12.225143] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 13:39:12.225199] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 13:39:12.225429] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:39:12.225465] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:39:12.225512] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 13:39:12.225712] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:39:12.225741] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:39:12.249721] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 13:39:12.264666] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:39:12.265141] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 13:39:12.269734] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:39:12.273187] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 13:39:12.284104] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:39:12.287162] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:39:12.287316] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:39:16.203058] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 13:39:16.212088] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:39:16.215633] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:39:16.215767] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:39:16.216103] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 13:39:16.219737] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:39:16.219791] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:39:16.221650] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 5259 >[2018-09-22 13:39:16.221774] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:39:16.225533] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 13:39:16.232659] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:39:16.236072] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:39:16.236204] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:39:16.238815] W [socket.c:600:__socket_rwv] 0-glustershd: readv on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket failed (No data available) >[2018-09-22 13:39:16.238857] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 13:39:17.222024] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:39:21.222785] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:39:21.222955] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped >[2018-09-22 13:39:21.222989] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped >[2018-09-22 13:39:21.223065] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:39:21.223089] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:39:21.223160] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:39:21.223183] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:39:28.430892] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:39:35.432207] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:39:42.433520] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:39:49.434941] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:39:56.436511] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:03.438173] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:08.278825] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:40:10.440161] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:17.441436] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:24.442885] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:31.444242] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:38.445746] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:45.447091] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:52.448463] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:40:59.449821] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:06.451228] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:13.452679] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:20.454881] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:27.456281] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:34.457764] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:41.459201] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:48.460541] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:41:55.461943] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:02.463359] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:09.464836] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:16.466196] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:23.467666] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:30.469108] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:42:31.274784] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x39545) [0x7f2ca345f545] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=gluster_vmstore --last=yes >[2018-09-22 13:42:31.618608] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x39545) [0x7f2ca345f545] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=gluster_vmstore --last=yes >[2018-09-22 13:42:31.618764] I [MSGID: 106542] [glusterd-utils.c:8228:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 3526 >[2018-09-22 13:42:32.645643] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:42:32.645697] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:42:32.646049] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 13:42:32.646087] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:42:32.646217] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:42:32.646247] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:42:32.646340] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:42:32.646388] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:42:50.606600] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 13:42:50.708060] E [MSGID: 106005] [glusterd-utils.c:6062:glusterd_brick_start] 0-management: Unable to start brick 10.0.14.12:/gluster-bricks/brick1/brick1 >[2018-09-22 13:42:50.734721] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 13:42:50.734732] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:42:50.734823] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 13:42:51.769586] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:42:51.769650] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:42:51.769728] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:42:51.769756] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:42:55.773623] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2ca350549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 13:42:56.252965] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2ca350549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 13:42:50.733472] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:42:50.733488] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:43:23.427535] W [socket.c:600:__socket_rwv] 0-management: readv on /var/run/gluster/09a0d99ac750f199f4d0b8e5b92ecfc3.socket failed (No data available) >[2018-09-22 13:43:23.436225] W [glusterd-handler.c:6064:__glusterd_brick_rpc_notify] 0-management: got disconnect from stale rpc on /gluster-bricks/brick1/brick1 >[2018-09-22 13:44:27.782735] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 13:44:27.890508] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49153 >[2018-09-22 13:44:27.891184] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:44:27.917089] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:44:27.917135] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:44:27.918437] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 5624 >[2018-09-22 13:44:27.932314] I [MSGID: 106006] [glusterd-svc-mgmt.c:328:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. >[2018-09-22 13:44:28.918719] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:44:32.919473] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:44:32.919739] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:44:32.919776] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:44:32.919851] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:44:32.919877] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:44:32.945735] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2ca350549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 13:44:33.461900] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2ca350549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 13:44:40.488630] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:44:47.489939] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:44:54.491558] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:45:01.493204] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:45:05.739764] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x39545) [0x7f2ca345f545] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=gluster_vmstore --last=yes >[2018-09-22 13:45:06.066884] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x39545) [0x7f2ca345f545] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2ca3504f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2caea05dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=gluster_vmstore --last=yes >[2018-09-22 13:45:06.067030] I [MSGID: 106542] [glusterd-utils.c:8228:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 5751 >[2018-09-22 13:45:08.494509] W [socket.c:3272:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4a62442afedbe58d404d2249840ea688.socket, (No such file or directory) >[2018-09-22 13:45:07.093079] I [MSGID: 106568] [glusterd-proc-mgmt.c:92:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 5624 >[2018-09-22 13:45:08.494690] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 13:45:08.494882] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 13:45:08.494914] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 13:45:08.494988] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 13:45:08.495016] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 13:45:08.495207] I [MSGID: 106144] [glusterd-pmap.c:396:pmap_registry_remove] 0-pmap: removing brick /gluster-bricks/brick1/brick1.rdma on port 49153 >[2018-09-22 13:45:08.495322] W [socket.c:600:__socket_rwv] 0-management: readv on /var/run/gluster/09a0d99ac750f199f4d0b8e5b92ecfc3.socket failed (No data available) >[2018-09-22 13:45:08.508097] W [socket.c:600:__socket_rwv] 0-socket.management: writev on 10.0.14.12:49151 failed (Broken pipe) >[2018-09-22 13:45:08.508140] I [socket.c:2481:socket_event_handler] 0-transport: EPOLLERR - disconnecting now >[2018-09-22 13:45:07.092707] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 13:45:07.092739] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 13:46:01.614276] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (No data available) >[2018-09-22 13:46:01.614370] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 13:46:01.614760] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f2ca34491ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f2ca3453128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f2ca350acf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 13:46:01.614783] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 13:46:10.180710] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 13:46:10.182996] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 13:46:12.507781] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.11:24007 failed (Connection refused); disconnecting socket >[2018-09-22 13:47:25.767583] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f2cad816e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x555ebcc714b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x555ebcc7132b] ) 0-: received signum (15), shutting down >[2018-09-22 13:50:29.446184] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 13:50:29.458089] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 13:50:29.458160] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 13:50:29.458178] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 13:50:35.365716] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 13:50:35.367354] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 13:50:35.588714] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 13:50:35.589023] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 13:50:35.589078] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 13:50:35.589119] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:50:35.589216] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 13:50:35.594770] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 13:50:35.594817] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 13:50:35.594906] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 13:50:35.600113] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 13:50:35.601579] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 13:50:35.604858] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 13:50:35.619118] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:50:35.623815] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 13:50:35.641628] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:50:35.644700] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 13:50:35.651034] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 13:50:35.654682] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:50:35.657702] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:50:35.657813] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 13:50:35.660995] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:50:35.665312] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:50:35.665425] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:50:35.665479] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:50:35.669140] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:50:35.669238] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:50:35.669298] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 13:50:35.672124] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 13:50:35.672232] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 13:50:35.672294] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:12:28.481963] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.11:24007 failed (No data available) >[2018-09-22 14:12:28.482016] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-11.ad.squeakz.net> (<f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 14:12:28.482452] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f1c84d7b1ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f1c84d85128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f1c84e3ccf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 14:12:28.482476] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 14:12:34.110147] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 14:12:34.118115] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:12:36.514933] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 14:12:36.521795] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:12:36.525084] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:12:38.609563] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 14:12:38.613593] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:12:38.617018] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:12:38.618522] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:20:51.360996] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f1c8f148e25] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x56396b15e4b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x6b) [0x56396b15e32b] ) 0-: received signum (15), shutting down >[2018-09-22 14:20:51.440777] I [MSGID: 100030] [glusterfsd.c:2511:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.12.14 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) >[2018-09-22 14:20:51.447590] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536 >[2018-09-22 14:20:51.447639] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory >[2018-09-22 14:20:51.447652] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory >[2018-09-22 14:20:56.796729] I [MSGID: 106513] [glusterd-store.c:2241:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202 >[2018-09-22 14:20:56.797748] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 86559324-725d-44a3-8d8a-09a1ca8ab243 >[2018-09-22 14:20:57.018128] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 14:20:57.018231] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 >[2018-09-22 14:20:57.018301] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 14:20:57.018352] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 14:20:57.018446] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >[2018-09-22 14:20:57.023673] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout >[2018-09-22 14:20:57.023709] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 14:20:57.023789] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction >Final graph: >+------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type socket,rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: >+------------------------------------------------------------------------------+ >[2018-09-22 14:20:57.028006] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 >[2018-09-22 14:21:01.140043] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 14:21:01.150569] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:21:01.635041] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a, host: ovirt-11.ad.squeakz.net, port: 0 >[2018-09-22 14:21:01.639112] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:21:01.651125] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:21:01.651234] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:21:01.651604] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 14:21:01.655119] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:21:01.659787] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-11.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 14:21:01.665764] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:21:01.668502] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:21:01.668602] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: f6c82f0c-c6e0-42de-82e0-e7b1ad633a1a >[2018-09-22 14:21:01.857620] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 14:21:01.861802] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:21:01.865161] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 14:21:01.871606] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:21:01.874969] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:21:01.877224] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:23:40.039737] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:23:40.042500] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:26:00.329679] W [socket.c:600:__socket_rwv] 0-management: readv on 10.0.10.13:24007 failed (Connection reset by peer) >[2018-09-22 14:26:00.329770] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-13.ad.squeakz.net> (<2eedbeab-db3d-40ad-8d6b-2a7963e647b4>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 14:26:00.330425] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f2b4d5291ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f2b4d533128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f2b4d5eacf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 14:26:00.330457] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 14:26:00.331628] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f2b58a98ebb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f2b5885dd9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f2b5885debe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7f2b5885f640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f2b58860130] ))))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(3)) called at 2018-09-22 14:24:40.351943 (xid=0xc) >[2018-09-22 14:26:00.331673] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on ovirt-13.ad.squeakz.net. Please check log file for details. >[2018-09-22 14:26:28.527067] E [socket.c:2376:socket_connect_finish] 0-management: connection to 10.0.10.13:24007 failed (No route to host); disconnecting socket >[2018-09-22 14:27:31.929603] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 14:27:31.940927] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:27:31.944861] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 14:27:31.951868] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 14:27:31.955635] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:27:31.958868] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:27:31.968120] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:27:31.971603] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:27:31.974235] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:27:31.977530] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:28:00.953248] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:28:00.955135] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:29:01.257706] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:29:01.260018] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:31:01.916293] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:31:01.918411] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:37:03.829440] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:37:03.831181] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:38:04.132757] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:38:04.134776] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:39:04.442397] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:39:04.444283] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:40:04.748845] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:40:04.750959] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:42:05.390694] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:42:05.392600] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:43:05.693395] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:43:05.695452] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:44:05.998642] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:44:06.000586] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:46:06.621923] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:46:06.623853] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:47:40.269924] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <ovirt-13.ad.squeakz.net> (<2eedbeab-db3d-40ad-8d6b-2a7963e647b4>), in state <Peer in Cluster>, has disconnected from glusterd. >[2018-09-22 14:47:40.270586] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x231ba) [0x7f2b4d5291ba] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0x2d128) [0x7f2b4d533128] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xe4cf5) [0x7f2b4d5eacf5] ) 0-management: Lock for vol gluster_vmstore not held >[2018-09-22 14:47:40.270617] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gluster_vmstore >[2018-09-22 14:47:45.869830] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202 >[2018-09-22 14:47:45.881618] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:47:50.721598] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to ovirt-13.ad.squeakz.net (0), ret: 0, op_ret: 0 >[2018-09-22 14:47:50.729163] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:47:50.732519] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:47:50.732703] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:47:50.742776] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4, host: ovirt-13.ad.squeakz.net, port: 0 >[2018-09-22 14:47:50.746329] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:47:50.749747] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend >[2018-09-22 14:47:50.753341] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2eedbeab-db3d-40ad-8d6b-2a7963e647b4 >[2018-09-22 14:49:07.541175] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:49:07.543366] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:50:07.849886] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:50:07.852128] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 14:53:08.771848] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 14:53:08.774307] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 15:05:12.496542] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:05:12.498516] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 15:06:12.817631] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:06:12.819522] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 15:07:13.139837] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:07:13.141507] E [MSGID: 106301] [glusterd-syncop.c:1353:gd_stage_op_phase] 0-management: Staging of operation 'Volume Status' failed on localhost : Volume gluster_vmstore is not started >[2018-09-22 15:08:03.855984] I [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a fresh brick process for brick /gluster-bricks/brick1/brick1 >[2018-09-22 15:08:04.010697] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /gluster-bricks/brick1/brick1.rdma on port 49152 >[2018-09-22 15:08:04.011393] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 >[2018-09-22 15:08:04.033817] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 >[2018-09-22 15:08:04.034033] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 >[2018-09-22 15:08:04.034163] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped >[2018-09-22 15:08:04.034189] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped >[2018-09-22 15:08:04.034630] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 >[2018-09-22 15:08:04.035679] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped >[2018-09-22 15:08:04.035718] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped >[2018-09-22 15:08:04.035753] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service >[2018-09-22 15:08:05.041235] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 >[2018-09-22 15:08:05.041617] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 >[2018-09-22 15:08:05.041805] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped >[2018-09-22 15:08:05.041851] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped >[2018-09-22 15:08:05.041909] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 >[2018-09-22 15:08:05.042092] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped >[2018-09-22 15:08:05.042116] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped >[2018-09-22 15:08:05.075556] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2b4d5e549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2b4d5e4f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2b58ae5dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 15:08:05.634058] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdf49a) [0x7f2b4d5e549a] -->/usr/lib64/glusterfs/3.12.14/xlator/mgmt/glusterd.so(+0xdef5d) [0x7f2b4d5e4f5d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f2b58ae5dd5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=gluster_vmstore --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd >[2018-09-22 15:08:13.447768] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:09:13.759986] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:16:15.990972] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:16:52.100401] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 15:16:52.100445] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 15:16:52.100614] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 15:16:57.697683] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 15:16:57.697683] and [2018-09-22 15:18:16.606740] >[2018-09-22 15:23:18.151686] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:25:18.781947] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:28:19.708449] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:29:20.037512] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:32:20.966504] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:33:21.271312] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:35:21.884622] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:40:23.410061] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:42:24.026066] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:44:24.653340] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:47:25.573801] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:49:26.199869] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:53:48.445804] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:53:48.765766] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 15:55:28.055435] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:03:30.526903] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:05:53.460807] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 16:05:53.460807] and [2018-09-22 16:06:31.471438] >[2018-09-22 16:07:31.780785] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:10:32.715324] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:15:34.277289] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:16:52.409616] I [MSGID: 106062] [glusterd-geo-rep.c:298:__glusterd_handle_gsync_set] 0-management: master not found, while handling geo-replication options >[2018-09-22 16:16:52.409664] I [MSGID: 106062] [glusterd-geo-rep.c:305:__glusterd_handle_gsync_set] 0-management: slave not found, while handling geo-replication options >[2018-09-22 16:16:52.410140] W [MSGID: 106062] [glusterd-geo-rep.c:2292:glusterd_op_gsync_args_get] 0-management: master not found >[2018-09-22 16:17:34.893134] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:18:35.207959] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:20:35.866668] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:23:36.816571] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:29:38.731648] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:31:39.379282] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:32:39.685137] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:38:41.661154] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:42:42.927104] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:45:43.863622] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:49:13.487558] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 16:49:13.487558] and [2018-09-22 16:50:45.421104] >[2018-09-22 16:52:46.026990] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:53:46.348912] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:56:47.284652] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 16:59:48.258293] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:04:49.775543] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:06:50.391008] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:07:44.153572] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 17:07:44.153572] and [2018-09-22 17:08:51.032125] >[2018-09-22 17:16:53.568369] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:17:53.882990] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:19:54.512903] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:23:55.787492] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:25:53.174976] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:25:53.497057] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:31:58.278640] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:34:59.235126] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:35:59.544563] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:36:59.868811] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:38:00.200308] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:43:01.855287] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 17:43:01.855287] and [2018-09-22 17:44:28.728123] >[2018-09-22 17:48:03.547201] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:52:04.828090] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:53:05.141349] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:55:05.770329] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 17:59:07.070320] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:02:08.032209] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:03:08.342359] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:05:08.989745] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:07:09.629402] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:08:09.937899] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:10:10.579868] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:11:10.913854] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:12:11.251468] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:15:53.158116] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >The message "I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore" repeated 2 times between [2018-09-22 18:15:53.158116] and [2018-09-22 18:16:12.535790] >[2018-09-22 18:17:12.842599] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:18:13.178289] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:22:14.425974] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:23:14.740734] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:25:15.387049] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:32:17.593082] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:34:35.207331] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:34:35.523031] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:36:18.846942] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:40:20.133997] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:41:20.456073] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:44:21.374118] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:45:21.681905] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:49:22.960515] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:50:23.273937] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:51:23.584188] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:53:28.636656] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:53:28.947493] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:55:24.946114] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:56:25.258953] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:57:25.566232] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:58:25.874728] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 18:59:26.181195] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:07:28.776414] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:08:29.087502] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:10:29.734872] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:14:31.051970] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:15:31.361573] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:17:32.007634] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:18:32.312042] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore >[2018-09-22 19:20:32.932231] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gluster_vmstore
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1624586
:
1486018
|
1486019
|
1486020
| 1486021 |
1486022
|
1486023
|
1503545