Attachment 1446229 Details for Bug 1584644 – glustershd.log

glustershd.log

glustershd.log (text/plain), 68.88 KB, created by Saravanakumar on 2018-05-31 11:27:50 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Saravanakumar

Created: 2018-05-31 11:27:50 UTC

Size: 68.88 KB

patch

obsolete

>sh-4.2# tailf -300 /var/log/glusterfs/glustershd.log 
>[2018-05-30 11:47:55.255780] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:47:39.247468] and [2018-05-30 11:47:55.257983]
>[2018-05-30 11:47:55.257984] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:47:59.257963] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:47:43.249720] and [2018-05-30 11:47:59.260117]
>[2018-05-30 11:47:59.260118] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:48:03.260372] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:47:47.252269] and [2018-05-30 11:48:03.262649]
>[2018-05-30 11:48:03.262653] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:48:07.262944] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:47:51.255408] and [2018-05-30 11:48:07.264944]
>[2018-05-30 11:48:07.265057] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:48:07.265086] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 2 times between [2018-05-30 11:47:59.260118] and [2018-05-30 11:48:07.265087]
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 3 times between [2018-05-30 11:47:55.257984] and [2018-05-30 11:48:07.265088]
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:47:51.255614] and [2018-05-30 11:48:07.265112]
>[2018-05-30 11:48:11.265478] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 11:48:11.267455] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 11:48:11.267584] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:48:11.267601] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:48:11.267607] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:48:11.267614] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:48:11.267643] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:48:15.267987] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 11:48:15.270142] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:48:15.270143] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:48:19.270633] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 2 times between [2018-05-30 11:48:11.267601] and [2018-05-30 11:48:19.272791]
>[2018-05-30 11:48:19.272793] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:48:23.272968] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 3 times between [2018-05-30 11:48:11.267584] and [2018-05-30 11:48:23.275041]
>[2018-05-30 11:48:23.275044] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:48:27.275195] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:11.267455] and [2018-05-30 11:48:27.277340]
>[2018-05-30 11:48:27.277483] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:48:31.278024] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:11.267643] and [2018-05-30 11:48:27.277543]
>[2018-05-30 11:48:31.280008] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:15.270143] and [2018-05-30 11:48:31.280155]
>[2018-05-30 11:48:31.280179] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:48:35.280373] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:19.272793] and [2018-05-30 11:48:35.282506]
>[2018-05-30 11:48:35.282507] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:48:39.282891] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:48:23.275044] and [2018-05-30 11:48:39.285066]
>[2018-05-30 11:48:39.285067] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:48:43.285184] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:48:27.277483] and [2018-05-30 11:48:43.287302]
>[2018-05-30 11:48:43.287305] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:48:47.287455] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:31.280008] and [2018-05-30 11:48:47.289476]
>[2018-05-30 11:48:47.289593] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:48:51.289703] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:31.280179] and [2018-05-30 11:48:47.289649]
>[2018-05-30 11:48:51.291828] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:35.282507] and [2018-05-30 11:48:51.291986]
>[2018-05-30 11:48:51.292011] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:48:55.292082] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:39.285067] and [2018-05-30 11:48:55.294344]
>[2018-05-30 11:48:55.294346] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:48:59.294273] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:48:43.287305] and [2018-05-30 11:48:59.296376]
>[2018-05-30 11:48:59.296378] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:49:03.296864] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:48:47.289593] and [2018-05-30 11:49:03.299041]
>[2018-05-30 11:49:03.299044] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:49:07.299343] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:48:51.291828] and [2018-05-30 11:49:07.301411]
>[2018-05-30 11:49:07.301530] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:49:11.301715] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:48:51.292011] and [2018-05-30 11:49:07.301588]
>[2018-05-30 11:49:11.303731] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:48:55.294346] and [2018-05-30 11:49:11.303891]
>[2018-05-30 11:49:11.303915] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:49:15.304531] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:48:59.296378] and [2018-05-30 11:49:15.306786]
>[2018-05-30 11:49:15.306787] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:49:19.307324] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:03.299044] and [2018-05-30 11:49:19.309472]
>[2018-05-30 11:49:19.309473] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:49:23.309264] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:07.301530] and [2018-05-30 11:49:23.311333]
>[2018-05-30 11:49:23.311336] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:49:27.311643] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:11.303731] and [2018-05-30 11:49:27.313592]
>[2018-05-30 11:49:27.313704] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:49:31.314201] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:11.303915] and [2018-05-30 11:49:27.313759]
>[2018-05-30 11:49:31.316174] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:49:15.306787] and [2018-05-30 11:49:31.316314]
>[2018-05-30 11:49:31.316337] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:49:35.316803] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:49:19.309473] and [2018-05-30 11:49:35.319003]
>[2018-05-30 11:49:35.319004] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:49:39.319170] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:23.311336] and [2018-05-30 11:49:39.321254]
>[2018-05-30 11:49:39.321255] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:49:43.321751] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:27.313704] and [2018-05-30 11:49:43.323845]
>[2018-05-30 11:49:43.323848] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:49:47.324123] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:31.316174] and [2018-05-30 11:49:47.326111]
>[2018-05-30 11:49:47.326222] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:49:51.326671] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:31.316337] and [2018-05-30 11:49:47.326279]
>[2018-05-30 11:49:51.328709] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:49:35.319004] and [2018-05-30 11:49:51.328881]
>[2018-05-30 11:49:51.328909] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:49:55.329064] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:49:39.321255] and [2018-05-30 11:49:55.331351]
>[2018-05-30 11:49:55.331353] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:49:59.332775] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:49:43.323848] and [2018-05-30 11:49:59.334868]
>[2018-05-30 11:49:59.334869] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:50:03.334115] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:49:47.326222] and [2018-05-30 11:50:03.336395]
>[2018-05-30 11:50:03.336399] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:50:07.336155] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:49:51.328709] and [2018-05-30 11:50:07.338137]
>[2018-05-30 11:50:07.338248] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:50:07.338277] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 2 times between [2018-05-30 11:49:59.334869] and [2018-05-30 11:50:07.338279]
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 3 times between [2018-05-30 11:49:55.331353] and [2018-05-30 11:50:07.338280]
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:49:51.328909] and [2018-05-30 11:50:07.338304]
>[2018-05-30 11:50:11.338835] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 11:50:11.340785] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 11:50:11.340925] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:50:11.340941] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:50:11.340948] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:50:11.340954] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:50:11.340982] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:50:15.341375] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 11:50:15.343632] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:50:15.343633] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:50:19.343704] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 2 times between [2018-05-30 11:50:11.340941] and [2018-05-30 11:50:19.345771]
>[2018-05-30 11:50:19.345773] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:50:23.346159] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 3 times between [2018-05-30 11:50:11.340925] and [2018-05-30 11:50:23.348518]
>[2018-05-30 11:50:23.348522] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:50:27.348293] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:50:11.340785] and [2018-05-30 11:50:27.350731]
>[2018-05-30 11:50:27.350865] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:50:31.350787] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:50:11.340982] and [2018-05-30 11:50:27.350932]
>[2018-05-30 11:50:31.352797] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:50:15.343633] and [2018-05-30 11:50:31.352948]
>[2018-05-30 11:50:31.352970] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:50:35.353508] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:50:19.345773] and [2018-05-30 11:50:35.355776]
>[2018-05-30 11:50:35.355777] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:50:39.355909] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:50:23.348522] and [2018-05-30 11:50:39.358007]
>[2018-05-30 11:50:39.358009] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:50:43.358577] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:50:27.350865] and [2018-05-30 11:50:43.360653]
>[2018-05-30 11:50:43.360656] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>[2018-05-30 11:50:47.363531] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 4 times between [2018-05-30 11:50:31.352797] and [2018-05-30 11:50:47.365553]
>[2018-05-30 11:50:47.365684] W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]
>[2018-05-30 11:50:51.363690] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 4 times between [2018-05-30 11:50:31.352970] and [2018-05-30 11:50:47.365739]
>[2018-05-30 11:50:51.365750] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 4 times between [2018-05-30 11:50:35.355777] and [2018-05-30 11:50:51.365907]
>[2018-05-30 11:50:51.365929] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:50:55.366395] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]" repeated 4 times between [2018-05-30 11:50:39.358009] and [2018-05-30 11:50:55.368600]
>[2018-05-30 11:50:55.368601] I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event
>[2018-05-30 11:50:59.368302] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]" repeated 4 times between [2018-05-30 11:50:43.360656] and [2018-05-30 11:50:59.370370]
>[2018-05-30 11:50:59.370372] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>[2018-05-30 11:51:03.372060] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>The message "W [MSGID: 114043] [client-handshake.c:1108:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to set the volume [No such file or directory]" repeated 4 times between [2018-05-30 11:50:47.365684] and [2018-05-30 11:51:03.374267]
>[2018-05-30 11:51:03.374270] W [MSGID: 114007] [client-handshake.c:1137:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]
>The message "I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available" repeated 3 times between [2018-05-30 11:50:51.365929] and [2018-05-30 11:51:03.374324]
>[2018-05-30 11:51:04.340725] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:51:08.378710] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 11:51:14.382384] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 11:52:02.627954] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 11:52:02.628038] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 11:52:02.628070] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:52:02.628094] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:52:07.278033] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0)
>[2018-05-30 11:52:07.280575] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 11:52:07.284922] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'.
>[2018-05-30 11:52:07.284948] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 11:52:07.285275] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1
>The message "I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)" repeated 3 times between [2018-05-30 11:50:51.365750] and [2018-05-30 11:51:03.374151]
>[2018-05-30 11:51:03.374298] E [MSGID: 114044] [client-handshake.c:1143:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: SETVOLUME on remote-host failed: remote-subvolume "/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick" is not found [No such file or directory]
>The message "I [MSGID: 114051] [client-handshake.c:1274:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: sending CHILD_CONNECTING event" repeated 2 times between [2018-05-30 11:50:55.368601] and [2018-05-30 11:51:03.374299]
>[2018-05-30 11:52:10.462038] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 11:52:10.464057] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 11:52:10.464414] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'.
>[2018-05-30 11:52:10.464426] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 11:52:10.464545] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1
>[2018-05-30 11:52:10.476870] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585
>[2018-05-30 11:52:10.533004] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_0.stat (a6393a88-fc71-4dda-b3bf-c1b5d2ae05b3) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 11:52:10.637920] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/global.stat (cc46db0c-4c9a-4760-bdb6-19cf6ffae32b) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 11:52:10.790061] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_12406.stat (6f7cbc6f-f5f3-485d-b18a-d51dc33cef73) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 11:52:10.835828] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 784fe888-9dfe-4b8d-8563-a0dca0952df0. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:10.838805] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_16385.stat (bf0f25fa-989c-4eef-a059-277f63064ded) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 11:52:10.844857] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:10.846634] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001
>[2018-05-30 11:52:10.940502] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.049080] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.050467] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d
>[2018-05-30 11:52:11.053996] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 1469f4b1-a945-4678-9a6e-598d195d0c5d. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.354021] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.355315] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c
>[2018-05-30 11:52:11.357516] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 11fea777-c8d6-4e95-ac2a-12e6db1d749c. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.408073] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.409247] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066
>[2018-05-30 11:52:11.411628] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 2c0a4933-f89f-4d21-9dbb-e2f8ae875066. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.494499] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b. sources=[0] 1  sinks=2 
>[2018-05-30 11:52:11.495687] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b
>[2018-05-30 11:52:11.497922] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 29b8c921-6afd-4cb6-998f-7ccf76d7576b. sources=[0] 1  sinks=2 
>[2018-05-30 11:54:45.538451] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:54:49.762471] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 11:54:58.912602] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-30 11:55:01.918650] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-30 12:11:02.631477] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 12:11:02.631569] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:11:02.744082] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 12:11:02.744127] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:11:07.032396] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0)
>[2018-05-30 12:11:07.035627] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 12:11:07.036802] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'.
>[2018-05-30 12:11:07.036824] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 12:11:07.040268] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1
>[2018-05-30 12:11:07.096229] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-heketidbstorage-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001
>[2018-05-30 12:11:07.294295] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-heketidbstorage-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[1] 2  sinks=0 
>[2018-05-30 12:11:10.857059] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 12:11:10.868104] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 12:11:10.879381] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'.
>[2018-05-30 12:11:10.879397] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 12:11:10.883183] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1
>[2018-05-30 12:11:10.942346] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c
>[2018-05-30 12:11:11.036153] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:11.131175] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c
>[2018-05-30 12:11:11.146357] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on a4ddd9e6-5f8e-4c6b-a583-fe1beaad4d74. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:12.689201] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c/postmaster.pid (473cb705-21ee-438f-8047-3b0e3ff0a41c) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 12:11:14.249914] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0f5d26ac-1f6c-486d-8bd2-4e7eab57ad9c. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:14.325138] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on b119e966-c973-4068-be74-6ab04d29339c
>[2018-05-30 12:11:20.401600] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on b119e966-c973-4068-be74-6ab04d29339c. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:20.540738] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 7858a277-7e09-4d22-ab02-2d42c07f5c14. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:20.541452] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585
>[2018-05-30 12:11:20.543465] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_0.stat (761f6e30-1557-46e8-9289-8e2b0d04df02) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 12:11:20.546000] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/global.stat (a0c9f594-87d6-4ccc-b8fa-30239da1268b) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 12:11:20.581339] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_12406.stat (65e42858-784e-42bf-9a80-6d9dff70345d) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 12:11:20.608519] W [MSGID: 108015] [afr-self-heal-entry.c:56:afr_selfheal_entry_delete] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: expunging file 0d399ac5-9c3b-4e15-8d72-225b3acbd585/db_16385.stat (1b8d1ba2-a2b9-4e9e-b5f1-2dcdb293d2c1) on vol_2aaf562cfd128cc6a174022f18b683b0-client-2
>[2018-05-30 12:11:20.649977] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 0d399ac5-9c3b-4e15-8d72-225b3acbd585. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:20.686552] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 25485600-c3e7-4911-8f30-3e4497462fd5
>[2018-05-30 12:11:20.691774] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 25485600-c3e7-4911-8f30-3e4497462fd5. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:20.694075] I [MSGID: 108026] [afr-self-heal-entry.c:848:afr_selfheal_entry_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing entry selfheal on 00000000-0000-0000-0000-000000000001
>[2018-05-30 12:11:20.790163] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed entry selfheal on 00000000-0000-0000-0000-000000000001. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.043306] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 6d4d6d40-9edd-480c-86f1-22a2d817ef29. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.165015] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 719172aa-ab30-406d-a38a-321d59940dd2. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.299508] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 69f57f3f-00bf-4263-8d58-a226b2597d1b. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.363388] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on ffa8f2d3-0872-407f-96f0-b0344a576360. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.440521] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 3730168a-bd8b-40e8-a718-6f2e8e4a3b9c. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.504656] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.506131] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e
>[2018-05-30 12:11:21.509161] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 29db6fb5-b2b6-487f-a24a-fa1041de538e. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.568228] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.569726] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2
>[2018-05-30 12:11:21.572838] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 08275b16-0f6f-4327-bf69-2211efd7afe2. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.661707] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.663063] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13
>[2018-05-30 12:11:21.666015] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 5c5dbc3f-fda1-4a1a-9e95-c9a350bbeb13. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.667588] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 69e972bc-01f7-402f-bec1-a7ea57b0f533
>[2018-05-30 12:11:21.670440] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 69e972bc-01f7-402f-bec1-a7ea57b0f533. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.711009] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.712935] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e
>[2018-05-30 12:11:21.717067] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 3c299278-4b2e-4e55-8e30-610ceebabe1e. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.767983] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on 536921c5-3a91-4cda-9982-3ca87e986188. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:21.769246] I [MSGID: 108026] [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: performing metadata selfheal on 536921c5-3a91-4cda-9982-3ca87e986188
>[2018-05-30 12:11:21.885931] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed metadata selfheal on 536921c5-3a91-4cda-9982-3ca87e986188. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:22.030169] I [MSGID: 108026] [afr-self-heal-common.c:1212:afr_log_selfheal] 0-vol_2aaf562cfd128cc6a174022f18b683b0-replicate-0: Completed data selfheal on f8a99ef3-044f-4231-baa6-3f118eae102c. sources=[0] 1  sinks=2 
>[2018-05-30 12:11:39.859233] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 12:11:40.858571] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 12:11:29.132019] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:11:29.132056] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:12:29.201296] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-heketidbstorage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 12:12:29.201370] E [MSGID: 114058] [client-handshake.c:1564:client_query_portmap_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2018-05-30 12:12:29.201402] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:12:29.201424] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-30 12:12:32.912545] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-heketidbstorage-client-0: changing port to 49152 (from 0)
>[2018-05-30 12:12:32.917553] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-heketidbstorage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 12:12:32.918819] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-heketidbstorage-client-0: Connected to heketidbstorage-client-0, attached to remote volume '/var/lib/heketi/mounts/vg_20815059c81012a55f4c4fa14725695f/brick_be8168fc1b53eee4a0d6d5c6ed26a098/brick'.
>[2018-05-30 12:12:32.918834] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-heketidbstorage-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 12:12:32.919465] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-heketidbstorage-client-0: Server lk version = 1
>[2018-05-30 12:12:36.935172] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: changing port to 49152 (from 0)
>[2018-05-30 12:12:36.937148] I [MSGID: 114057] [client-handshake.c:1477:select_server_supported_programs] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2018-05-30 12:12:36.937538] I [MSGID: 114046] [client-handshake.c:1230:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Connected to vol_2aaf562cfd128cc6a174022f18b683b0-client-2, attached to remote volume '/var/lib/heketi/mounts/vg_3aa9dfed04dfbc76fba92a6256a0d294/brick_f1ed81a1533f0e92e25d94006ec8a8ad/brick'.
>[2018-05-30 12:12:36.937549] I [MSGID: 114047] [client-handshake.c:1241:client_setvolume_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2018-05-30 12:12:36.937666] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: Server lk version = 1
>[2018-05-31 10:28:18.978476] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-heketidbstorage-client-0: disconnected from heketidbstorage-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-31 10:28:22.946576] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: disconnected from vol_2aaf562cfd128cc6a174022f18b683b0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
>[2018-05-31 10:28:30.562722] E [socket.c:2360:socket_connect_finish] 0-heketidbstorage-client-0: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-31 10:28:33.568540] E [socket.c:2360:socket_connect_finish] 0-vol_2aaf562cfd128cc6a174022f18b683b0-client-2: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>

Actions: View

Attachments on bug 1584644: 1446228 | 1446229