Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1446229 Details for
Bug 1584644
Gluster mountpoint doesn't fall back immediately to other gluster nodes when the node serving the volume is disconnected/shutdown
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
glustershd.log
glustershd.log (text/plain), 68.88 KB, created by
Saravanakumar
on 2018-05-31 11:27:50 UTC
(
hide
)
Description:
glustershd.log
Filename:
MIME Type:
Creator:
Saravanakumar
Created:
2018-05-31 11:27:50 UTC
Size:
68.88 KB
patch
obsolete
>sh-4.2# tailf -300 /var/log/glusterfs/glustershd.log >[2018-05-30 11:47:55.255780] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:47:39.247468] and [2018-05-30 11:47:55.257983] >[2018-05-30 11:47:55.257984] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:47:59.257963] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:47:43.249720] and [2018-05-30 11:47:59.260117] >[2018-05-30 11:47:59.260118] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:48:03.260372] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:47:47.252269] and [2018-05-30 11:48:03.262649] >[2018-05-30 11:48:03.262653] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:48:07.262944] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:47:51.255408] and [2018-05-30 11:48:07.264944] >[2018-05-30 11:48:07.265057] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:48:07.265086] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 2 times between [2018-05-30 11:47:59.260118] and [2018-05-30 11:48:07.265087] >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 3 times between [2018-05-30 11:47:55.257984] and [2018-05-30 11:48:07.265088] >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:47:51.255614] and [2018-05-30 11:48:07.265112] >[2018-05-30 11:48:11.265478] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 11:48:11.267455] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 11:48:11.267584] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:48:11.267601] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:48:11.267607] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:48:11.267614] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:48:11.267643] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:48:15.267987] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 11:48:15.270142] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:48:15.270143] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:48:19.270633] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 2 times between [2018-05-30 11:48:11.267601] and [2018-05-30 11:48:19.272791] >[2018-05-30 11:48:19.272793] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:48:23.272968] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 3 times between [2018-05-30 11:48:11.267584] and [2018-05-30 11:48:23.275041] >[2018-05-30 11:48:23.275044] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:48:27.275195] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:11.267455] and [2018-05-30 11:48:27.277340] >[2018-05-30 11:48:27.277483] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:48:31.278024] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:11.267643] and [2018-05-30 11:48:27.277543] >[2018-05-30 11:48:31.280008] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:15.270143] and [2018-05-30 11:48:31.280155] >[2018-05-30 11:48:31.280179] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:48:35.280373] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:19.272793] and [2018-05-30 11:48:35.282506] >[2018-05-30 11:48:35.282507] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:48:39.282891] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:48:23.275044] and [2018-05-30 11:48:39.285066] >[2018-05-30 11:48:39.285067] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:48:43.285184] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:48:27.277483] and [2018-05-30 11:48:43.287302] >[2018-05-30 11:48:43.287305] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:48:47.287455] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:31.280008] and [2018-05-30 11:48:47.289476] >[2018-05-30 11:48:47.289593] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:48:51.289703] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:31.280179] and [2018-05-30 11:48:47.289649] >[2018-05-30 11:48:51.291828] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:35.282507] and [2018-05-30 11:48:51.291986] >[2018-05-30 11:48:51.292011] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:48:55.292082] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:39.285067] and [2018-05-30 11:48:55.294344] >[2018-05-30 11:48:55.294346] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:48:59.294273] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:48:43.287305] and [2018-05-30 11:48:59.296376] >[2018-05-30 11:48:59.296378] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:49:03.296864] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:48:47.289593] and [2018-05-30 11:49:03.299041] >[2018-05-30 11:49:03.299044] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:49:07.299343] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:51.291828] and [2018-05-30 11:49:07.301411] >[2018-05-30 11:49:07.301530] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:49:11.301715] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:51.292011] and [2018-05-30 11:49:07.301588] >[2018-05-30 11:49:11.303731] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:55.294346] and [2018-05-30 11:49:11.303891] >[2018-05-30 11:49:11.303915] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:49:15.304531] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:59.296378] and [2018-05-30 11:49:15.306786] >[2018-05-30 11:49:15.306787] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:49:19.307324] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:03.299044] and [2018-05-30 11:49:19.309472] >[2018-05-30 11:49:19.309473] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:49:23.309264] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:07.301530] and [2018-05-30 11:49:23.311333] >[2018-05-30 11:49:23.311336] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:49:27.311643] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:11.303731] and [2018-05-30 11:49:27.313592] >[2018-05-30 11:49:27.313704] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:49:31.314201] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:11.303915] and [2018-05-30 11:49:27.313759] >[2018-05-30 11:49:31.316174] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:49:15.306787] and [2018-05-30 11:49:31.316314] >[2018-05-30 11:49:31.316337] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:49:35.316803] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:49:19.309473] and [2018-05-30 11:49:35.319003] >[2018-05-30 11:49:35.319004] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:49:39.319170] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:23.311336] and [2018-05-30 11:49:39.321254] >[2018-05-30 11:49:39.321255] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:49:43.321751] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:27.313704] and [2018-05-30 11:49:43.323845] >[2018-05-30 11:49:43.323848] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:49:47.324123] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:31.316174] and [2018-05-30 11:49:47.326111] >[2018-05-30 11:49:47.326222] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:49:51.326671] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:31.316337] and [2018-05-30 11:49:47.326279] >[2018-05-30 11:49:51.328709] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:49:35.319004] and [2018-05-30 11:49:51.328881] >[2018-05-30 11:49:51.328909] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:49:55.329064] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:49:39.321255] and [2018-05-30 11:49:55.331351] >[2018-05-30 11:49:55.331353] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:49:59.332775] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:43.323848] and [2018-05-30 11:49:59.334868] >[2018-05-30 11:49:59.334869] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:50:03.334115] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:47.326222] and [2018-05-30 11:50:03.336395] >[2018-05-30 11:50:03.336399] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:50:07.336155] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:51.328709] and [2018-05-30 11:50:07.338137] >[2018-05-30 11:50:07.338248] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:50:07.338277] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 2 times between [2018-05-30 11:49:59.334869] and [2018-05-30 11:50:07.338279] >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 3 times between [2018-05-30 11:49:55.331353] and [2018-05-30 11:50:07.338280] >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:51.328909] and [2018-05-30 11:50:07.338304] >[2018-05-30 11:50:11.338835] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 11:50:11.340785] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 11:50:11.340925] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:50:11.340941] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:50:11.340948] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:50:11.340954] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:50:11.340982] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:50:15.341375] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 11:50:15.343632] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:50:15.343633] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:50:19.343704] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 2 times between [2018-05-30 11:50:11.340941] and [2018-05-30 11:50:19.345771] >[2018-05-30 11:50:19.345773] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:50:23.346159] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 3 times between [2018-05-30 11:50:11.340925] and [2018-05-30 11:50:23.348518] >[2018-05-30 11:50:23.348522] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:50:27.348293] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:50:11.340785] and [2018-05-30 11:50:27.350731] >[2018-05-30 11:50:27.350865] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:50:31.350787] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:50:11.340982] and [2018-05-30 11:50:27.350932] >[2018-05-30 11:50:31.352797] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:50:15.343633] and [2018-05-30 11:50:31.352948] >[2018-05-30 11:50:31.352970] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:50:35.353508] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:50:19.345773] and [2018-05-30 11:50:35.355776] >[2018-05-30 11:50:35.355777] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:50:39.355909] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:50:23.348522] and [2018-05-30 11:50:39.358007] >[2018-05-30 11:50:39.358009] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:50:43.358577] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:50:27.350865] and [2018-05-30 11:50:43.360653] >[2018-05-30 11:50:43.360656] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >[2018-05-30 11:50:47.363531] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:50:31.352797] and [2018-05-30 11:50:47.365553] >[2018-05-30 11:50:47.365684] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory] >[2018-05-30 11:50:51.363690] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:50:31.352970] and [2018-05-30 11:50:47.365739] >[2018-05-30 11:50:51.365750] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:50:35.355777] and [2018-05-30 11:50:51.365907] >[2018-05-30 11:50:51.365929] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:50:55.366395] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:50:39.358009] and [2018-05-30 11:50:55.368600] >[2018-05-30 11:50:55.368601] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event >[2018-05-30 11:50:59.368302] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:50:43.360656] and [2018-05-30 11:50:59.370370] >[2018-05-30 11:50:59.370372] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >[2018-05-30 11:51:03.372060] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:50:47.365684] and [2018-05-30 11:51:03.374267] >[2018-05-30 11:51:03.374270] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument] >The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 3 times between [2018-05-30 11:50:51.365929] and [2018-05-30 11:51:03.374324] >[2018-05-30 11:51:04.340725] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:51:08.378710] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket >[2018-05-30 11:51:14.382384] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket >[2018-05-30 11:52:02.627954] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 11:52:02.628038] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 11:52:02.628070] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:52:02.628094] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:52:07.278033] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0) >[2018-05-30 11:52:07.280575] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 11:52:07.284922] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'. >[2018-05-30 11:52:07.284948] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 11:52:07.285275] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1 >The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 3 times between [2018-05-30 11:50:51.365750] and [2018-05-30 11:51:03.374151] >[2018-05-30 11:51:03.374298] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory] >The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 2 times between [2018-05-30 11:50:55.368601] and [2018-05-30 11:51:03.374299] >[2018-05-30 11:52:10.462038] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 11:52:10.464057] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 11:52:10.464414] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'. >[2018-05-30 11:52:10.464426] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 11:52:10.464545] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1 >[2018-05-30 11:52:10.476870] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585 >[2018-05-30 11:52:10.533004] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_0.stat (a6393a88-fc71-4dda-b3bf-c1b5d2ae05b3) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 11:52:10.637920] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/global.stat (cc46db0c-4c9a-4760-bdb6-19cf6ffae32b) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 11:52:10.790061] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_12406.stat (6f7cbc6f-f5f3-485d-b18a-d51dc33cef73) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 11:52:10.835828] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 784fe888-9dfe-4b8d-8563-a0dca0952df0. sources=[0] 1 sinks=2 >[2018-05-30 11:52:10.838805] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_16385.stat (bf0f25fa-989c-4eef-a059-277f63064ded) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 11:52:10.844857] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585. sources=[0] 1 sinks=2 >[2018-05-30 11:52:10.846634] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001 >[2018-05-30 11:52:10.940502] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.049080] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.050467] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d >[2018-05-30 11:52:11.053996] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.354021] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.355315] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c >[2018-05-30 11:52:11.357516] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.408073] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.409247] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066 >[2018-05-30 11:52:11.411628] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.494499] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b. sources=[0] 1 sinks=2 >[2018-05-30 11:52:11.495687] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b >[2018-05-30 11:52:11.497922] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b. sources=[0] 1 sinks=2 >[2018-05-30 11:54:45.538451] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:54:49.762471] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 11:54:58.912602] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket >[2018-05-30 11:55:01.918650] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket >[2018-05-30 12:11:02.631477] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 12:11:02.631569] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:11:02.744082] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 12:11:02.744127] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:11:07.032396] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0) >[2018-05-30 12:11:07.035627] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 12:11:07.036802] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'. >[2018-05-30 12:11:07.036824] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 12:11:07.040268] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1 >[2018-05-30 12:11:07.096229] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-heketidbstorage-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001 >[2018-05-30 12:11:07.294295] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-heketidbstorage-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[1] 2 sinks=0 >[2018-05-30 12:11:10.857059] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 12:11:10.868104] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 12:11:10.879381] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'. >[2018-05-30 12:11:10.879397] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 12:11:10.883183] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1 >[2018-05-30 12:11:10.942346] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c >[2018-05-30 12:11:11.036153] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c. sources=[0] 1 sinks=2 >[2018-05-30 12:11:11.131175] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c >[2018-05-30 12:11:11.146357] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on a4ddd9e6-5f8e-4c6b-a583-fe1beaad4d74. sources=[0] 1 sinks=2 >[2018-05-30 12:11:12.689201] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c/postmaster.pid (473cb705-21ee-438f-8047-3b0e3ff0a41c) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 12:11:14.249914] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c. sources=[0] 1 sinks=2 >[2018-05-30 12:11:14.325138] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on b119e966-c973-4068-be74-6ab04d29339c >[2018-05-30 12:11:20.401600] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on b119e966-c973-4068-be74-6ab04d29339c. sources=[0] 1 sinks=2 >[2018-05-30 12:11:20.540738] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 7858a277-7e09-4d22-ab02-2d42c07f5c14. sources=[0] 1 sinks=2 >[2018-05-30 12:11:20.541452] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585 >[2018-05-30 12:11:20.543465] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_0.stat (761f6e30-1557-46e8-9289-8e2b0d04df02) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 12:11:20.546000] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/global.stat (a0c9f594-87d6-4ccc-b8fa-30239da1268b) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 12:11:20.581339] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_12406.stat (65e42858-784e-42bf-9a80-6d9dff70345d) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 12:11:20.608519] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_16385.stat (1b8d1ba2-a2b9-4e9e-b5f1-2dcdb293d2c1) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2 >[2018-05-30 12:11:20.649977] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585. sources=[0] 1 sinks=2 >[2018-05-30 12:11:20.686552] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 25485600-c3e7-4911-8f30-3e4497462fd5 >[2018-05-30 12:11:20.691774] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 25485600-c3e7-4911-8f30-3e4497462fd5. sources=[0] 1 sinks=2 >[2018-05-30 12:11:20.694075] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001 >[2018-05-30 12:11:20.790163] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.043306] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 6d4d6d40-9edd-480c-86f1-22a2d817ef29. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.165015] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 719172aa-ab30-406d-a38a-321d59940dd2. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.299508] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 69f57f3f-00bf-4263-8d58-a226b2597d1b. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.363388] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on ffa8f2d3-0872-407f-96f0-b0344a576360. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.440521] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 3730168a-bd8b-40e8-a718-6f2e8e4a3b9c. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.504656] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.506131] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e >[2018-05-30 12:11:21.509161] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.568228] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.569726] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2 >[2018-05-30 12:11:21.572838] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.661707] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.663063] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13 >[2018-05-30 12:11:21.666015] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.667588] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 69e972bc-01f7-402f-bec1-a7ea57b0f533 >[2018-05-30 12:11:21.670440] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 69e972bc-01f7-402f-bec1-a7ea57b0f533. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.711009] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.712935] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e >[2018-05-30 12:11:21.717067] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.767983] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 536921c5-3a91-4cda-9982-3ca87e986188. sources=[0] 1 sinks=2 >[2018-05-30 12:11:21.769246] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 536921c5-3a91-4cda-9982-3ca87e986188 >[2018-05-30 12:11:21.885931] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 536921c5-3a91-4cda-9982-3ca87e986188. sources=[0] 1 sinks=2 >[2018-05-30 12:11:22.030169] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on f8a99ef3-044f-4231-baa6-3f118eae102c. sources=[0] 1 sinks=2 >[2018-05-30 12:11:39.859233] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket >[2018-05-30 12:11:40.858571] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket >[2018-05-30 12:11:29.132019] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:11:29.132056] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:12:29.201296] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 12:12:29.201370] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2018-05-30 12:12:29.201402] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:12:29.201424] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-30 12:12:32.912545] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0) >[2018-05-30 12:12:32.917553] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 12:12:32.918819] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'. >[2018-05-30 12:12:32.918834] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 12:12:32.919465] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1 >[2018-05-30 12:12:36.935172] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0) >[2018-05-30 12:12:36.937148] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2018-05-30 12:12:36.937538] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'. >[2018-05-30 12:12:36.937549] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2018-05-30 12:12:36.937666] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1 >[2018-05-31 10:28:18.978476] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-31 10:28:22.946576] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2018-05-31 10:28:30.562722] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket >[2018-05-31 10:28:33.568540] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1584644
:
1446228
| 1446229