Attachment 588419 Details for Bug 827405 – Client log file.

Client log file.

root-client.log (text/x-log), 1005.78 KB, created by Vijaykumar Koppad on 2012-06-01 11:37:49 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Vijaykumar Koppad

Created: 2012-06-01 11:37:49 UTC

Size: 1005.78 KB

patch

obsolete

>[2012-06-01 03:57:44.687411] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa45
>[2012-06-01 03:57:44.707685] E [glusterfsd-mgmt.c:1555:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
>[2012-06-01 03:57:44.707776] E [glusterfsd-mgmt.c:1623:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:doa)
>[2012-06-01 03:57:44.708020] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb6) [0x3b4180f516] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x3b4180f302] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x298) [0x40c348]))) 0-: received signum (0), shutting down
>[2012-06-01 03:57:44.708132] I [fuse-bridge.c:4643:fini] 0-fuse: Unmounting '/root/client'.
>[2012-06-01 03:57:58.155693] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa45
>[2012-06-01 03:57:58.173405] I [io-cache.c:1549:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104233984
>[2012-06-01 03:57:58.173555] I [io-cache.c:1549:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104233984
>[2012-06-01 03:57:58.194193] I [client.c:2142:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 03:57:58.199347] I [client.c:2142:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 03:57:58.204278] I [client.c:2142:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 03:57:58.209216] I [client.c:2142:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 03:57:58.215571] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 03:57:58.215712] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 03:57:58.215844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 03:57:58.216022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 03:58:02.169524] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 03:58:02.169947] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 03:58:02.169981] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 03:58:02.170061] I [afr-common.c:3627:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 03:58:02.170212] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-06-01 03:58:02.174970] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 03:58:02.175285] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 03:58:02.175316] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 03:58:02.176605] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-06-01 03:58:02.179798] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 03:58:02.180102] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 03:58:02.180131] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 03:58:02.180203] I [afr-common.c:3627:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 03:58:02.180336] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-06-01 03:58:02.184607] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 03:58:02.185052] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 03:58:02.185083] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 03:58:02.199890] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-06-01 03:58:02.200123] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-06-01 03:58:02.200396] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
>[2012-06-01 03:58:02.201091] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-06-01 03:58:02.201849] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-06-01 04:00:02.498962] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:00:05.108168] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2012-06-01 04:16:46.779298] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:16:47.899066] I [io-cache.c:1549:check_cache_size_ok] 1-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:16:47.899189] I [io-cache.c:1549:check_cache_size_ok] 1-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:16:47.899290] I [quota.c:3085:quota_parse_limits] 1-master-quota: no "limit-set" option provided
>[2012-06-01 04:16:47.977016] I [client.c:2142:notify] 1-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:47.992436] I [client.c:2142:notify] 1-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:48.011814] I [client.c:2142:notify] 1-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:48.016562] I [client.c:2142:notify] 1-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:16:48.031955] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:16:48.032062] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:16:48.032147] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:16:48.032230] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:16:49.993769] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:16:51.027484] I [io-cache.c:1549:check_cache_size_ok] 2-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:16:51.027600] I [io-cache.c:1549:check_cache_size_ok] 2-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:16:51.038352] I [client.c:2142:notify] 2-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:51.046975] I [client.c:2142:notify] 2-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:51.063556] I [client.c:2142:notify] 2-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:51.077667] I [client.c:2142:notify] 2-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:16:51.089880] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:16:51.090349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:16:51.090852] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:16:51.090981] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:16:51.371313] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:51.371763] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:16:51.371796] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:51.371919] I [afr-common.c:3627:afr_notify] 1-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:16:51.372076] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-0: Server lk version = 1
>[2012-06-01 04:16:51.376478] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:51.376822] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:16:51.376854] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:51.377262] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-1: Server lk version = 1
>[2012-06-01 04:16:51.382279] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:51.382709] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:16:51.382747] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:51.382853] I [afr-common.c:3627:afr_notify] 1-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:16:51.382938] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-2: Server lk version = 1
>[2012-06-01 04:16:51.387490] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:51.387905] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:16:51.387937] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:51.399939] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 1
>[2012-06-01 04:16:51.400060] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-3: Server lk version = 1
>[2012-06-01 04:16:53.218357] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:16:54.329266] I [io-cache.c:1549:check_cache_size_ok] 3-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:16:54.329390] I [io-cache.c:1549:check_cache_size_ok] 3-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:16:54.329505] I [quota.c:3085:quota_parse_limits] 3-master-quota: no "limit-set" option provided
>[2012-06-01 04:16:54.373906] I [client.c:2142:notify] 3-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:54.383603] I [client.c:2142:notify] 3-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:54.431894] I [client.c:2142:notify] 3-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:54.455705] I [client.c:2142:notify] 3-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:16:54.483504] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:54.484842] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:54.486479] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:54.487573] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:16:54.487829] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:16:54.487892] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:54.490010] I [afr-common.c:3627:afr_notify] 2-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:16:54.490074] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:16:54.490210] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:16:54.490240] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:54.492163] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:16:54.492417] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:16:54.492448] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:54.492658] I [afr-common.c:3627:afr_notify] 2-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:16:54.492711] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:16:54.492836] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:54.493145] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-0: Server lk version = 1
>[2012-06-01 04:16:54.493204] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-1: Server lk version = 1
>[2012-06-01 04:16:54.493253] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-2: Server lk version = 1
>[2012-06-01 04:16:54.493312] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:16:54.493336] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:54.506279] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 2
>[2012-06-01 04:16:54.507484] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-3: Server lk version = 1
>[2012-06-01 04:16:56.437676] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:16:57.469105] I [io-cache.c:1549:check_cache_size_ok] 4-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:16:57.469260] I [io-cache.c:1549:check_cache_size_ok] 4-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:16:57.491542] I [client.c:2142:notify] 4-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:57.499254] I [client.c:2142:notify] 4-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:57.516879] I [client.c:2142:notify] 4-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:16:57.524054] I [client.c:2142:notify] 4-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:16:57.532017] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:16:57.532263] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:16:57.532393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:16:57.532587] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:16:58.488513] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:58.488842] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:16:58.488904] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:58.488988] I [afr-common.c:3627:afr_notify] 3-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:16:58.489130] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-0: Server lk version = 1
>[2012-06-01 04:16:58.495933] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:58.496233] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:16:58.496264] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:58.496614] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-1: Server lk version = 1
>[2012-06-01 04:16:58.504902] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:58.505251] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:16:58.505284] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:58.505378] I [afr-common.c:3627:afr_notify] 3-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:16:58.505463] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-2: Server lk version = 1
>[2012-06-01 04:16:58.511818] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:16:58.512133] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:16:58.512163] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:16:58.520610] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 3
>[2012-06-01 04:16:58.520700] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-3: Server lk version = 1
>[2012-06-01 04:16:59.671090] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:16:59.726935] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 3-master-replicate-0: added root inode
>[2012-06-01 04:16:59.728193] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 3-master-replicate-1: added root inode
>[2012-06-01 04:16:59.728473] I [client.c:2151:notify] 0-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:16:59.728769] I [client.c:2090:client_rpc_notify] 0-master-client-0: disconnected
>[2012-06-01 04:16:59.728807] I [client.c:2151:notify] 0-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:16:59.729017] I [client.c:2090:client_rpc_notify] 0-master-client-1: disconnected
>[2012-06-01 04:16:59.729044] E [afr-common.c:3664:afr_notify] 0-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:16:59.729071] I [client.c:2151:notify] 0-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:16:59.729331] I [client.c:2090:client_rpc_notify] 0-master-client-2: disconnected
>[2012-06-01 04:16:59.729370] I [client.c:2151:notify] 0-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:16:59.729594] I [client.c:2090:client_rpc_notify] 0-master-client-3: disconnected
>[2012-06-01 04:16:59.729650] E [afr-common.c:3664:afr_notify] 0-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:01.518092] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:01.518261] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:01.518421] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:01.518443] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:01.518567] I [afr-common.c:3627:afr_notify] 4-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:01.518601] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:01.518633] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:01.518897] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-0: Server lk version = 1
>[2012-06-01 04:17:01.518922] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-1: Server lk version = 1
>[2012-06-01 04:17:01.535050] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:01.535289] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:01.535309] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:01.535355] I [afr-common.c:3627:afr_notify] 4-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:17:01.535561] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-2: Server lk version = 1
>[2012-06-01 04:17:01.547135] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:01.547300] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:01.547318] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:01.555111] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 4
>[2012-06-01 04:17:01.556083] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-3: Server lk version = 1
>[2012-06-01 04:17:01.557738] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 4-master-replicate-0: added root inode
>[2012-06-01 04:17:01.559243] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 4-master-replicate-1: added root inode
>[2012-06-01 04:17:01.561167] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8522070e-0759-48a4-897b-8b3018f587e0)
>[2012-06-01 04:17:01.563676] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4e9ef3d5-fab8-4169-80b7-e6cbc2d798a5)
>[2012-06-01 04:17:01.566271] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:17c95f95-37e1-4ae1-898f-f64d4d4f0bbc)
>[2012-06-01 04:17:01.568284] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:de0af784-3a82-4504-bb61-3a2e4880f172)
>[2012-06-01 04:17:01.572632] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e22b122f-a193-4fae-83f4-14c7f805e1c7)
>[2012-06-01 04:17:01.575349] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9877cbfe-cf9b-4d11-8070-0d5eec833f9b)
>[2012-06-01 04:17:01.578231] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:2796ef76-229d-4d3a-aceb-310a82684fdb)
>[2012-06-01 04:17:01.580389] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d9a23130-a7c3-431e-8398-282a8b4a083a)
>[2012-06-01 04:17:01.583641] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:5f5cc251-41a9-4d27-a77a-3a8cb8fabbdf)
>[2012-06-01 04:17:01.586505] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f2031c96-8c98-4c06-8963-7f980665e88a)
>[2012-06-01 04:17:01.590049] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:04649108-a2c6-48d1-a62f-7847350d311f)
>[2012-06-01 04:17:01.592927] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:3e92e90c-9959-46fa-87a3-f29e1cb32158)
>[2012-06-01 04:17:01.595687] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:7d49129a-acb5-4b81-b56a-73532c46a36b)
>[2012-06-01 04:17:01.598705] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:962d7a36-4f82-4dd3-9fc7-92be4ef55549)
>[2012-06-01 04:17:01.601817] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:032921e9-9e6f-4468-b48b-f67cc8d1c4c7)
>[2012-06-01 04:17:01.604429] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:478c58a9-a0d2-4920-94c3-845aef639a4e)
>[2012-06-01 04:17:01.607739] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b57e5fb2-1b88-47c5-bba7-9a8fd56eae4f)
>[2012-06-01 04:17:01.610596] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:ba221d26-f588-4f8d-af77-c8940b63e797)
>[2012-06-01 04:17:01.613661] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f3910016-2214-42fc-a9b8-7cf3b2c896b8)
>[2012-06-01 04:17:01.617380] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:044a8cdc-2180-459f-b048-f1049910ffc2)
>[2012-06-01 04:17:01.619785] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f428c0bc-6194-4cd3-956d-676d323429e8)
>[2012-06-01 04:17:01.621947] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a4b59479-af1f-4a69-9a5d-116a49047b7b)
>[2012-06-01 04:17:01.624524] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:bba7d9dd-596b-48d4-ae5f-e71a91d7e34f)
>[2012-06-01 04:17:01.627282] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8ad59a9b-7b83-4ef0-a2b5-ec0df7616c1d)
>[2012-06-01 04:17:01.629543] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9e50920b-78b8-49aa-8889-7976dbc3ea80)
>[2012-06-01 04:17:01.632500] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4f58cedd-dd70-406d-a818-a99ed956b248)
>[2012-06-01 04:17:01.635978] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:563e9bfd-8c7e-4dbd-8657-85115a5ce75c)
>[2012-06-01 04:17:01.638649] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:c2bd75ce-c798-4a34-9f60-e32b9ef3b250)
>[2012-06-01 04:17:01.642217] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:32e09d06-6e72-458d-81da-0befb2e7e160)
>[2012-06-01 04:17:01.644989] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:561277dc-03e2-4127-baee-b556fc9799f3)
>[2012-06-01 04:17:01.648016] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e30db81a-ae6d-449b-a1c1-ba228ddf1da3)
>[2012-06-01 04:17:01.651338] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0e1cf4e7-4ec0-4550-8d0f-c94046418fc4)
>[2012-06-01 04:17:01.654320] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d3f0ce18-5bf7-4b93-a3c4-3caf801b20de)
>[2012-06-01 04:17:01.657209] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b77136c4-6e01-4db2-b5d5-dbf780d29375)
>[2012-06-01 04:17:01.659886] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:efbd2ddc-037a-4951-a85f-5de8826810a0)
>[2012-06-01 04:17:01.662492] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8085d7f1-27c8-4cb3-8543-e874db6331b1)
>[2012-06-01 04:17:01.665157] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8e70a386-2f45-4bef-be5f-c6569a0a0a49)
>[2012-06-01 04:17:01.667787] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:cd07e6d5-73d1-4a16-a904-21c930e98701)
>[2012-06-01 04:17:01.670779] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d7746c28-3d8b-4ae5-bef0-544df2868b25)
>[2012-06-01 04:17:01.673147] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:5b58fab9-d2fb-4492-9ff0-e0d627cd9fa0)
>[2012-06-01 04:17:01.675505] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e368c3c4-9b39-45c6-89a8-aa019c499002)
>[2012-06-01 04:17:01.678379] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:becfff3d-933c-43bb-a7a9-ec4f76041138)
>[2012-06-01 04:17:01.680715] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:3da1b304-76f6-4573-945c-e708c6642e66)
>[2012-06-01 04:17:01.683306] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a9b8a98e-ef6d-4b0f-be31-fca72cb624b5)
>[2012-06-01 04:17:01.686571] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:15d64ae4-84d6-47ed-a370-33ee91dd7f92)
>[2012-06-01 04:17:01.689232] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d55db600-df01-40a6-852b-f66cd5332de7)
>[2012-06-01 04:17:01.692181] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:2431005a-eb7c-4f4b-a371-a2aad1955d87)
>[2012-06-01 04:17:01.694919] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:1f0811fa-61fa-44a3-ab10-f3056af73053)
>[2012-06-01 04:17:01.698076] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9ee41d89-5c9c-4e7f-b520-f08c1f300ffb)
>[2012-06-01 04:17:01.701090] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:35b797aa-872a-45e0-992f-6c0a67000ff3)
>[2012-06-01 04:17:01.704066] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:54f59351-8320-4797-9c8b-a4a331898cbf)
>[2012-06-01 04:17:01.706750] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9e73e512-c3c1-444a-89a4-97a067332105)
>[2012-06-01 04:17:01.711075] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4c355011-989c-45a2-9457-833c6e46e40f)
>[2012-06-01 04:17:01.717953] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:58397717-8352-4bf1-b139-98b8ede2f7bb)
>[2012-06-01 04:17:01.720571] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8dbd1880-d485-4e32-a7cc-1a31b75cc066)
>[2012-06-01 04:17:01.723412] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4de155a8-c85c-4731-953f-60a6ff185495)
>[2012-06-01 04:17:01.725901] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a205824e-0f70-44a8-92f1-866792d9b745)
>[2012-06-01 04:17:01.728428] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f9643bbd-82db-467d-9c81-7980608f1f0e)
>[2012-06-01 04:17:01.731076] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e8aa16cd-95b3-44bd-a0b0-be8ddab1090f)
>[2012-06-01 04:17:01.733462] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:cf20a194-40aa-48a1-8269-c404d78ab3a3)
>[2012-06-01 04:17:01.737083] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:24f4f8d4-1d0d-4530-b8a3-bfc24aecc4b0)
>[2012-06-01 04:17:01.739368] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:95ac252d-f0af-4e81-b603-3f874fccdfb7)
>[2012-06-01 04:17:01.742035] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e02daf6d-11be-49a9-92b4-0efc44ae6c6c)
>[2012-06-01 04:17:01.744525] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:23843325-21f3-4669-82c5-fa30c1eee6fb)
>[2012-06-01 04:17:01.746875] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0f4ae84c-8e56-4f09-a397-87a7d210cc75)
>[2012-06-01 04:17:01.749312] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:6edeed61-f4ea-420c-b32d-dc461e950f58)
>[2012-06-01 04:17:01.751652] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e42acc3f-d1b0-4377-afa2-612fc4559d33)
>[2012-06-01 04:17:01.753860] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b6bdf08d-87e0-48b6-a675-ca45022bed58)
>[2012-06-01 04:17:01.756288] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:23b92615-8b00-4f7d-919b-dd30d836b438)
>[2012-06-01 04:17:01.759792] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0b979563-23c4-4715-a48c-3b22ae6aa32e)
>[2012-06-01 04:17:01.761458] I [client.c:2151:notify] 3-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:01.761499] I [client.c:2151:notify] 3-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:01.761535] I [client.c:2151:notify] 3-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:01.761561] I [client.c:2151:notify] 3-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:01.771447] I [client.c:2090:client_rpc_notify] 3-master-client-0: disconnected
>[2012-06-01 04:17:01.771491] I [client.c:2090:client_rpc_notify] 3-master-client-1: disconnected
>[2012-06-01 04:17:01.771503] E [afr-common.c:3664:afr_notify] 3-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:01.771529] I [client.c:2090:client_rpc_notify] 3-master-client-2: disconnected
>[2012-06-01 04:17:01.771557] I [client.c:2090:client_rpc_notify] 3-master-client-3: disconnected
>[2012-06-01 04:17:01.771568] E [afr-common.c:3664:afr_notify] 3-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:10.793972] I [io-cache.c:1549:check_cache_size_ok] 5-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:10.794083] I [io-cache.c:1549:check_cache_size_ok] 5-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:10.794146] I [quota.c:3085:quota_parse_limits] 5-master-quota: no "limit-set" option provided
>[2012-06-01 04:17:10.807828] I [client.c:2142:notify] 5-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:10.821847] I [client.c:2142:notify] 5-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:10.826423] I [client.c:2142:notify] 5-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:10.839797] I [client.c:2142:notify] 5-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:10.848932] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:10.849037] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:10.849652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:10.849751] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:17:14.561062] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:14.561660] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:14.561691] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:14.561871] I [afr-common.c:3627:afr_notify] 5-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:14.563211] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-0: Server lk version = 1
>[2012-06-01 04:17:14.567154] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:14.567456] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:14.567486] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:14.568141] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-1: Server lk version = 1
>[2012-06-01 04:17:14.578330] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:14.579779] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:14.579813] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:14.579912] I [afr-common.c:3627:afr_notify] 5-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:17:14.581027] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-2: Server lk version = 1
>[2012-06-01 04:17:14.581074] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:14.581921] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:14.581954] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:14.587230] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 5
>[2012-06-01 04:17:14.588460] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-3: Server lk version = 1
>[2012-06-01 04:17:14.589303] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 5-master-replicate-0: added root inode
>[2012-06-01 04:17:14.590543] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 5-master-replicate-1: added root inode
>[2012-06-01 04:17:14.629461] I [client.c:2151:notify] 4-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:14.629549] I [client.c:2151:notify] 4-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:14.629598] I [client.c:2151:notify] 4-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:14.629671] I [client.c:2151:notify] 4-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:14.633823] I [client.c:2090:client_rpc_notify] 4-master-client-0: disconnected
>[2012-06-01 04:17:14.633926] I [client.c:2090:client_rpc_notify] 4-master-client-1: disconnected
>[2012-06-01 04:17:14.633948] E [afr-common.c:3664:afr_notify] 4-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:14.633982] I [client.c:2090:client_rpc_notify] 4-master-client-2: disconnected
>[2012-06-01 04:17:14.634019] I [client.c:2090:client_rpc_notify] 4-master-client-3: disconnected
>[2012-06-01 04:17:14.634037] E [afr-common.c:3664:afr_notify] 4-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:18.007973] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:17:29.070840] I [io-cache.c:1549:check_cache_size_ok] 6-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:29.070935] I [io-cache.c:1549:check_cache_size_ok] 6-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:29.083663] I [client.c:2142:notify] 6-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:29.091641] I [client.c:2142:notify] 6-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:29.097010] I [client.c:2142:notify] 6-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:29.101539] I [client.c:2142:notify] 6-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:29.110780] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:29.110923] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:29.111006] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:29.111107] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:17:31.338187] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:17:32.477787] I [io-cache.c:1549:check_cache_size_ok] 7-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:32.477874] I [io-cache.c:1549:check_cache_size_ok] 7-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:32.477933] I [quota.c:3085:quota_parse_limits] 7-master-quota: no "limit-set" option provided
>[2012-06-01 04:17:32.665235] I [client.c:2142:notify] 7-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:32.674794] I [client.c:2142:notify] 7-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:32.683919] I [client.c:2142:notify] 7-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:32.686893] I [client.c:2142:notify] 7-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:32.695306] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:32.695691] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:32.701682] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:32.701948] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:32.703247] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:32.703269] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:32.703403] I [afr-common.c:3627:afr_notify] 6-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:32.703444] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:32.703457] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:32.703815] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:32.703839] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:32.703966] I [afr-common.c:3627:afr_notify] 6-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:17:32.704002] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:32.704016] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:32.746000] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 6
>[2012-06-01 04:17:32.746103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:32.746624] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-0: Server lk version = 1
>[2012-06-01 04:17:32.746713] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-1: Server lk version = 1
>[2012-06-01 04:17:32.746739] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-2: Server lk version = 1
>[2012-06-01 04:17:32.746759] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-3: Server lk version = 1
>[2012-06-01 04:17:32.746795] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:17:32.746945] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:32.747029] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:32.750812] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 6-master-replicate-0: added root inode
>[2012-06-01 04:17:32.752575] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 6-master-replicate-1: added root inode
>[2012-06-01 04:17:32.769110] I [client.c:2151:notify] 5-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:32.769165] I [client.c:2151:notify] 5-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:32.769194] I [client.c:2151:notify] 5-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:32.769224] I [client.c:2151:notify] 5-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:32.771163] I [client.c:2090:client_rpc_notify] 5-master-client-0: disconnected
>[2012-06-01 04:17:32.771359] I [client.c:2090:client_rpc_notify] 5-master-client-1: disconnected
>[2012-06-01 04:17:32.771376] E [afr-common.c:3664:afr_notify] 5-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:32.771399] I [client.c:2090:client_rpc_notify] 5-master-client-2: disconnected
>[2012-06-01 04:17:32.771421] I [client.c:2090:client_rpc_notify] 5-master-client-3: disconnected
>[2012-06-01 04:17:32.771432] E [afr-common.c:3664:afr_notify] 5-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:34.891133] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:17:35.937661] I [io-cache.c:1549:check_cache_size_ok] 8-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:35.937780] I [io-cache.c:1549:check_cache_size_ok] 8-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:35.945468] I [client.c:2142:notify] 8-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:35.952804] I [client.c:2142:notify] 8-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:35.957686] I [client.c:2142:notify] 8-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:35.962700] I [client.c:2142:notify] 8-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:35.974803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:35.974984] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:35.975112] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:35.975236] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:17:36.686607] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:36.694161] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:36.694674] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:36.694901] I [afr-common.c:3627:afr_notify] 7-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:36.700605] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-0: Server lk version = 1
>[2012-06-01 04:17:36.714261] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:36.714555] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:36.716469] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:36.716503] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:36.719223] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-1: Server lk version = 1
>[2012-06-01 04:17:36.720271] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:36.720596] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:36.720782] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:36.721306] I [afr-common.c:3627:afr_notify] 7-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:17:36.723026] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:36.723063] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:36.733079] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 7
>[2012-06-01 04:17:36.733175] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-3: Server lk version = 1
>[2012-06-01 04:17:36.734654] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-2: Server lk version = 1
>[2012-06-01 04:17:36.738713] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 7-master-replicate-0: added root inode
>[2012-06-01 04:17:36.741365] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 7-master-replicate-1: added root inode
>[2012-06-01 04:17:36.807314] I [client.c:2151:notify] 6-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:36.807426] I [client.c:2151:notify] 6-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:36.807478] I [client.c:2151:notify] 6-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:36.807522] I [client.c:2151:notify] 6-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:36.807524] I [client.c:2090:client_rpc_notify] 6-master-client-0: disconnected
>[2012-06-01 04:17:36.807633] I [client.c:2090:client_rpc_notify] 6-master-client-1: disconnected
>[2012-06-01 04:17:36.807656] E [afr-common.c:3664:afr_notify] 6-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:36.807689] I [client.c:2090:client_rpc_notify] 6-master-client-2: disconnected
>[2012-06-01 04:17:36.807722] I [client.c:2090:client_rpc_notify] 6-master-client-3: disconnected
>[2012-06-01 04:17:36.807757] E [afr-common.c:3664:afr_notify] 6-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:38.141204] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:17:39.718927] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:39.719263] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:39.719283] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:39.719417] I [afr-common.c:3627:afr_notify] 8-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:39.719730] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-0: Server lk version = 1
>[2012-06-01 04:17:39.723966] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:39.725516] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:39.725537] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:39.726996] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-1: Server lk version = 1
>[2012-06-01 04:17:39.730607] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:39.733368] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:39.733625] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:39.733742] I [afr-common.c:3627:afr_notify] 8-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:17:39.734987] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-2: Server lk version = 1
>[2012-06-01 04:17:39.741719] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:39.742249] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:39.742269] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:39.748880] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 8
>[2012-06-01 04:17:39.749985] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-3: Server lk version = 1
>[2012-06-01 04:17:39.750825] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 8-master-replicate-0: added root inode
>[2012-06-01 04:17:39.752460] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 8-master-replicate-1: added root inode
>[2012-06-01 04:17:39.757971] I [client.c:2151:notify] 7-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:39.758448] I [client.c:2151:notify] 7-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:39.758495] I [client.c:2151:notify] 7-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:39.758736] I [client.c:2151:notify] 7-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:39.759592] I [client.c:2090:client_rpc_notify] 7-master-client-2: disconnected
>[2012-06-01 04:17:39.760200] I [client.c:2090:client_rpc_notify] 7-master-client-0: disconnected
>[2012-06-01 04:17:39.760237] I [client.c:2090:client_rpc_notify] 7-master-client-1: disconnected
>[2012-06-01 04:17:39.760250] E [afr-common.c:3664:afr_notify] 7-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:39.760357] I [client.c:2090:client_rpc_notify] 7-master-client-3: disconnected
>[2012-06-01 04:17:39.760384] E [afr-common.c:3664:afr_notify] 7-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:44.247948] I [io-cache.c:1549:check_cache_size_ok] 9-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:44.248066] I [io-cache.c:1549:check_cache_size_ok] 9-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:44.248159] I [quota.c:3085:quota_parse_limits] 9-master-quota: no "limit-set" option provided
>[2012-06-01 04:17:44.255116] I [client.c:2142:notify] 9-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:44.261170] I [client.c:2142:notify] 9-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:44.266096] I [client.c:2142:notify] 9-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:44.270793] I [client.c:2142:notify] 9-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:44.281246] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:44.281390] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:44.281514] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:44.281652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:17:47.751058] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:47.752491] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:17:47.752525] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:47.752744] I [afr-common.c:3627:afr_notify] 9-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:17:47.755796] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-0: Server lk version = 1
>[2012-06-01 04:17:47.765936] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:47.768692] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:17:47.768736] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:47.769117] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:47.769576] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-1: Server lk version = 1
>[2012-06-01 04:17:47.770085] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:17:47.770113] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:47.770292] I [afr-common.c:3627:afr_notify] 9-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:17:47.770642] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-2: Server lk version = 1
>[2012-06-01 04:17:47.771011] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:17:47.772227] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:17:47.772261] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:17:47.777882] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 9
>[2012-06-01 04:17:47.779245] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-3: Server lk version = 1
>[2012-06-01 04:17:47.780941] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 9-master-replicate-0: added root inode
>[2012-06-01 04:17:47.782401] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 9-master-replicate-1: added root inode
>[2012-06-01 04:17:47.803473] I [client.c:2151:notify] 8-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:47.803536] I [client.c:2151:notify] 8-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:47.803576] I [client.c:2151:notify] 8-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:47.803615] I [client.c:2151:notify] 8-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:17:47.809174] I [client.c:2090:client_rpc_notify] 8-master-client-0: disconnected
>[2012-06-01 04:17:47.809226] I [client.c:2090:client_rpc_notify] 8-master-client-1: disconnected
>[2012-06-01 04:17:47.809247] E [afr-common.c:3664:afr_notify] 8-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:47.809282] I [client.c:2090:client_rpc_notify] 8-master-client-2: disconnected
>[2012-06-01 04:17:47.809318] I [client.c:2090:client_rpc_notify] 8-master-client-3: disconnected
>[2012-06-01 04:17:47.809337] E [afr-common.c:3664:afr_notify] 8-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:17:56.410701] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:17:57.447507] I [io-cache.c:1549:check_cache_size_ok] 10-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:17:57.447609] I [io-cache.c:1549:check_cache_size_ok] 10-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:17:57.455377] I [client.c:2142:notify] 10-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:57.459579] I [client.c:2142:notify] 10-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:57.463830] I [client.c:2142:notify] 10-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:17:57.468138] I [client.c:2142:notify] 10-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:17:57.479939] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:17:57.480083] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:17:57.480201] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:17:57.480314] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:00.779049] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:00.781420] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:00.781453] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:00.781549] I [afr-common.c:3627:afr_notify] 10-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:00.782298] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-0: Server lk version = 1
>[2012-06-01 04:18:00.789068] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:00.792927] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:00.792962] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:00.793847] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-1: Server lk version = 1
>[2012-06-01 04:18:00.795305] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:00.796262] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:00.796295] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:00.796474] I [afr-common.c:3627:afr_notify] 10-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:00.797265] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-2: Server lk version = 1
>[2012-06-01 04:18:00.801531] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:00.802979] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:00.803012] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:00.809427] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 10
>[2012-06-01 04:18:00.810150] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-3: Server lk version = 1
>[2012-06-01 04:18:00.813187] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 10-master-replicate-0: added root inode
>[2012-06-01 04:18:00.814345] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 10-master-replicate-1: added root inode
>[2012-06-01 04:18:00.821239] W [quota.c:2377:quota_fsync_cbk] 9-master-quota: quota context not set in inode (gfid:df187b59-ec7a-46a2-bf61-078ade2e92c9)
>[2012-06-01 04:18:00.826075] W [quota.c:2377:quota_fsync_cbk] 9-master-quota: quota context not set in inode (gfid:e5eb198b-c20d-4a32-96a8-ad473e6321a3)
>[2012-06-01 04:18:00.829116] I [client.c:2151:notify] 9-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:00.829202] I [client.c:2151:notify] 9-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:00.829249] I [client.c:2151:notify] 9-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:00.829262] I [client.c:2090:client_rpc_notify] 9-master-client-0: disconnected
>[2012-06-01 04:18:00.829912] I [client.c:2090:client_rpc_notify] 9-master-client-1: disconnected
>[2012-06-01 04:18:00.829944] E [afr-common.c:3664:afr_notify] 9-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:00.829306] I [client.c:2151:notify] 9-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:00.830059] I [client.c:2090:client_rpc_notify] 9-master-client-2: disconnected
>[2012-06-01 04:18:00.830600] I [client.c:2090:client_rpc_notify] 9-master-client-3: disconnected
>[2012-06-01 04:18:00.830659] E [afr-common.c:3664:afr_notify] 9-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:09.665125] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:10.899068] I [io-cache.c:1549:check_cache_size_ok] 11-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:10.899138] I [io-cache.c:1549:check_cache_size_ok] 11-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:10.899205] I [quota.c:3085:quota_parse_limits] 11-master-quota: no "limit-set" option provided
>[2012-06-01 04:18:10.905775] I [client.c:2142:notify] 11-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:10.908908] I [client.c:2142:notify] 11-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:10.912840] I [client.c:2142:notify] 11-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:10.915513] I [client.c:2142:notify] 11-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:10.924832] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:10.924931] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:10.925011] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:10.925101] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:14.824357] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:14.824637] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:14.826298] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:14.826331] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:14.826530] I [afr-common.c:3627:afr_notify] 11-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:14.826585] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:14.826606] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:14.829726] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-0: Server lk version = 1
>[2012-06-01 04:18:14.829773] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-1: Server lk version = 1
>[2012-06-01 04:18:14.910783] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:14.912618] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:14.912651] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:14.931516] I [afr-common.c:3627:afr_notify] 11-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:14.931676] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-2: Server lk version = 1
>[2012-06-01 04:18:14.959170] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:14.961193] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:14.961265] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:15.015400] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 11
>[2012-06-01 04:18:15.017774] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-3: Server lk version = 1
>[2012-06-01 04:18:15.035232] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 11-master-replicate-0: added root inode
>[2012-06-01 04:18:15.037483] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 11-master-replicate-1: added root inode
>[2012-06-01 04:18:15.055019] I [client.c:2151:notify] 10-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:15.055083] I [client.c:2151:notify] 10-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:15.055114] I [client.c:2151:notify] 10-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:15.055128] I [client.c:2090:client_rpc_notify] 10-master-client-0: disconnected
>[2012-06-01 04:18:15.055153] I [client.c:2151:notify] 10-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:15.057972] I [client.c:2090:client_rpc_notify] 10-master-client-1: disconnected
>[2012-06-01 04:18:15.057993] E [afr-common.c:3664:afr_notify] 10-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:15.058026] I [client.c:2090:client_rpc_notify] 10-master-client-2: disconnected
>[2012-06-01 04:18:15.058051] I [client.c:2090:client_rpc_notify] 10-master-client-3: disconnected
>[2012-06-01 04:18:15.058062] E [afr-common.c:3664:afr_notify] 10-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:18.523604] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:19.608540] I [io-cache.c:1549:check_cache_size_ok] 12-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:19.608677] I [io-cache.c:1549:check_cache_size_ok] 12-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:19.615294] I [client.c:2142:notify] 12-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:19.619718] I [client.c:2142:notify] 12-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:19.623898] I [client.c:2142:notify] 12-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:19.628375] I [client.c:2142:notify] 12-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:19.647670] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:19.647898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:19.648019] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:19.648135] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:22.959586] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:22.962288] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:22.962325] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:22.962523] I [afr-common.c:3627:afr_notify] 12-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:22.964080] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-0: Server lk version = 1
>[2012-06-01 04:18:22.968471] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:22.968998] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:22.969032] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:22.969996] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-1: Server lk version = 1
>[2012-06-01 04:18:22.970820] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:22.972432] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:22.972467] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:22.972567] I [afr-common.c:3627:afr_notify] 12-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:22.972917] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-2: Server lk version = 1
>[2012-06-01 04:18:22.979931] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:22.982197] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:22.982234] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:22.987153] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 12
>[2012-06-01 04:18:22.987549] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-3: Server lk version = 1
>[2012-06-01 04:18:22.990499] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 12-master-replicate-0: added root inode
>[2012-06-01 04:18:22.993330] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 12-master-replicate-1: added root inode
>[2012-06-01 04:18:23.013678] W [quota.c:2377:quota_fsync_cbk] 11-master-quota: quota context not set in inode (gfid:54e7852e-353d-4fa3-9173-40b186a45a4a)
>[2012-06-01 04:18:23.022215] I [client.c:2151:notify] 11-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:23.022305] I [client.c:2151:notify] 11-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:23.022357] I [client.c:2151:notify] 11-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:23.022388] I [client.c:2090:client_rpc_notify] 11-master-client-0: disconnected
>[2012-06-01 04:18:23.022435] I [client.c:2151:notify] 11-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:23.026174] I [client.c:2090:client_rpc_notify] 11-master-client-1: disconnected
>[2012-06-01 04:18:23.026209] E [afr-common.c:3664:afr_notify] 11-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:23.026250] I [client.c:2090:client_rpc_notify] 11-master-client-2: disconnected
>[2012-06-01 04:18:23.026290] I [client.c:2090:client_rpc_notify] 11-master-client-3: disconnected
>[2012-06-01 04:18:23.026310] E [afr-common.c:3664:afr_notify] 11-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:36.807714] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:37.899306] I [io-cache.c:1549:check_cache_size_ok] 13-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:37.899425] I [io-cache.c:1549:check_cache_size_ok] 13-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:37.899541] I [quota.c:3085:quota_parse_limits] 13-master-quota: no "limit-set" option provided
>[2012-06-01 04:18:37.906343] I [client.c:2142:notify] 13-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:37.911385] I [client.c:2142:notify] 13-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:37.915962] I [client.c:2142:notify] 13-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:37.920609] I [client.c:2142:notify] 13-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:37.931293] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:37.931482] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:37.931613] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:37.931742] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:40.093672] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:40.987060] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:40.992203] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:40.992234] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:40.992407] I [afr-common.c:3627:afr_notify] 13-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:40.992675] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-0: Server lk version = 1
>[2012-06-01 04:18:41.001955] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:41.003827] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:41.003859] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:41.005790] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-1: Server lk version = 1
>[2012-06-01 04:18:41.007010] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:41.009918] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:41.009952] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:41.010125] I [afr-common.c:3627:afr_notify] 13-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:41.012172] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-2: Server lk version = 1
>[2012-06-01 04:18:41.014865] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:41.019597] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:41.019655] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:41.024071] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 13
>[2012-06-01 04:18:41.024283] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-3: Server lk version = 1
>[2012-06-01 04:18:41.028211] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 13-master-replicate-0: added root inode
>[2012-06-01 04:18:41.030720] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 13-master-replicate-1: added root inode
>[2012-06-01 04:18:41.057809] I [client.c:2151:notify] 12-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:41.057916] I [client.c:2151:notify] 12-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:41.057965] I [client.c:2151:notify] 12-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:41.057968] I [client.c:2090:client_rpc_notify] 12-master-client-0: disconnected
>[2012-06-01 04:18:41.058028] I [client.c:2151:notify] 12-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:41.062828] I [client.c:2090:client_rpc_notify] 12-master-client-1: disconnected
>[2012-06-01 04:18:41.062855] E [afr-common.c:3664:afr_notify] 12-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:41.062924] I [client.c:2090:client_rpc_notify] 12-master-client-2: disconnected
>[2012-06-01 04:18:41.062960] I [client.c:2090:client_rpc_notify] 12-master-client-3: disconnected
>[2012-06-01 04:18:41.062978] E [afr-common.c:3664:afr_notify] 12-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:41.140520] I [io-cache.c:1549:check_cache_size_ok] 14-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:41.140621] I [io-cache.c:1549:check_cache_size_ok] 14-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:41.149044] I [client.c:2142:notify] 14-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:41.153286] I [client.c:2142:notify] 14-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:41.157258] I [client.c:2142:notify] 14-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:41.161255] I [client.c:2142:notify] 14-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:41.168625] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:41.168757] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:41.168897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:41.169010] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:43.629894] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:44.809724] I [io-cache.c:1549:check_cache_size_ok] 15-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:44.809802] I [io-cache.c:1549:check_cache_size_ok] 15-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:44.809862] I [quota.c:3085:quota_parse_limits] 15-master-quota: no "limit-set" option provided
>[2012-06-01 04:18:44.822705] I [client.c:2142:notify] 15-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:44.830296] I [client.c:2142:notify] 15-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:44.835847] I [client.c:2142:notify] 15-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:44.840980] I [client.c:2142:notify] 15-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:44.849122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:44.849205] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:44.850124] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:44.850200] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:45.016859] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:45.018335] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:45.018357] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:45.018485] I [afr-common.c:3627:afr_notify] 14-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:45.019886] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-0: Server lk version = 1
>[2012-06-01 04:18:45.027840] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:45.030573] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:45.030595] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:45.031643] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:45.033279] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-1: Server lk version = 1
>[2012-06-01 04:18:45.033461] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:45.036592] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:45.036636] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:45.036814] I [afr-common.c:3627:afr_notify] 14-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:45.037213] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-2: Server lk version = 1
>[2012-06-01 04:18:45.037247] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:45.038030] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:45.055261] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 14
>[2012-06-01 04:18:45.056649] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-3: Server lk version = 1
>[2012-06-01 04:18:45.058056] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 14-master-replicate-0: added root inode
>[2012-06-01 04:18:45.059204] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 14-master-replicate-1: added root inode
>[2012-06-01 04:18:45.084738] W [quota.c:2377:quota_fsync_cbk] 13-master-quota: quota context not set in inode (gfid:d518acaa-c775-41f8-98e3-610ff24709aa)
>[2012-06-01 04:18:45.094453] I [client.c:2151:notify] 13-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:45.094519] I [client.c:2151:notify] 13-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:45.094561] I [client.c:2151:notify] 13-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:45.094604] I [client.c:2151:notify] 13-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:45.094557] I [client.c:2090:client_rpc_notify] 13-master-client-0: disconnected
>[2012-06-01 04:18:45.097502] I [client.c:2090:client_rpc_notify] 13-master-client-1: disconnected
>[2012-06-01 04:18:45.097516] E [afr-common.c:3664:afr_notify] 13-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:45.097547] I [client.c:2090:client_rpc_notify] 13-master-client-2: disconnected
>[2012-06-01 04:18:45.097578] I [client.c:2090:client_rpc_notify] 13-master-client-3: disconnected
>[2012-06-01 04:18:45.097635] E [afr-common.c:3664:afr_notify] 13-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:48.035951] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:48.037647] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:48.040040] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:48.040132] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:48.041313] I [afr-common.c:3627:afr_notify] 15-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:18:48.041351] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:48.041365] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:48.042990] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-1: Server lk version = 1
>[2012-06-01 04:18:48.043017] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-0: Server lk version = 1
>[2012-06-01 04:18:48.050864] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:48.051722] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:48.051747] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:48.051798] I [afr-common.c:3627:afr_notify] 15-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:48.052464] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-2: Server lk version = 1
>[2012-06-01 04:18:48.067787] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:48.069159] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:48.069180] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:48.144970] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 15
>[2012-06-01 04:18:48.145323] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-3: Server lk version = 1
>[2012-06-01 04:18:48.146456] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 15-master-replicate-0: added root inode
>[2012-06-01 04:18:48.147069] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 15-master-replicate-1: added root inode
>[2012-06-01 04:18:48.165252] I [client.c:2151:notify] 14-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:48.165303] I [client.c:2151:notify] 14-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:48.165332] I [client.c:2151:notify] 14-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:48.165357] I [client.c:2151:notify] 14-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:48.170446] I [client.c:2090:client_rpc_notify] 14-master-client-0: disconnected
>[2012-06-01 04:18:48.170496] I [client.c:2090:client_rpc_notify] 14-master-client-1: disconnected
>[2012-06-01 04:18:48.170511] E [afr-common.c:3664:afr_notify] 14-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:48.170535] I [client.c:2090:client_rpc_notify] 14-master-client-2: disconnected
>[2012-06-01 04:18:48.170558] I [client.c:2090:client_rpc_notify] 14-master-client-3: disconnected
>[2012-06-01 04:18:48.170569] E [afr-common.c:3664:afr_notify] 14-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:52.065078] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:53.120492] I [io-cache.c:1549:check_cache_size_ok] 16-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:53.120603] I [io-cache.c:1549:check_cache_size_ok] 16-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:53.127418] I [client.c:2142:notify] 16-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:53.131787] I [client.c:2142:notify] 16-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:53.136282] I [client.c:2142:notify] 16-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:53.140509] I [client.c:2142:notify] 16-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:53.149313] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:53.149408] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:53.149464] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:53.151232] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:55.809665] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:18:57.003130] I [io-cache.c:1549:check_cache_size_ok] 17-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:18:57.003254] I [io-cache.c:1549:check_cache_size_ok] 17-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:18:57.003343] I [quota.c:3085:quota_parse_limits] 17-master-quota: no "limit-set" option provided
>[2012-06-01 04:18:57.026201] I [client.c:2142:notify] 17-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:57.030307] I [client.c:2142:notify] 17-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:57.043456] I [client.c:2142:notify] 17-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:18:57.047614] I [client.c:2142:notify] 17-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:18:57.062607] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:18:57.064991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:18:57.065108] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:18:57.065217] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:18:57.102416] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:57.108136] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:18:57.108170] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:57.108353] I [afr-common.c:3627:afr_notify] 16-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:18:57.111099] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-0: Server lk version = 1
>[2012-06-01 04:18:57.111163] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:57.115552] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:18:57.115586] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:57.121070] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-1: Server lk version = 1
>[2012-06-01 04:18:57.121334] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:57.127382] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:18:57.127419] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:57.127615] I [afr-common.c:3627:afr_notify] 16-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:18:57.128799] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-2: Server lk version = 1
>[2012-06-01 04:18:57.137942] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:18:57.141232] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:18:57.141266] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:18:57.229420] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 16
>[2012-06-01 04:18:57.231190] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-3: Server lk version = 1
>[2012-06-01 04:18:57.234175] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 16-master-replicate-0: added root inode
>[2012-06-01 04:18:57.236939] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 16-master-replicate-1: added root inode
>[2012-06-01 04:18:57.261543] I [client.c:2151:notify] 15-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:57.261645] I [client.c:2151:notify] 15-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:57.261692] I [client.c:2151:notify] 15-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:57.261737] I [client.c:2151:notify] 15-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:18:57.267768] I [client.c:2090:client_rpc_notify] 15-master-client-0: disconnected
>[2012-06-01 04:18:57.267823] I [client.c:2090:client_rpc_notify] 15-master-client-1: disconnected
>[2012-06-01 04:18:57.267844] E [afr-common.c:3664:afr_notify] 15-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:57.267912] I [client.c:2090:client_rpc_notify] 15-master-client-2: disconnected
>[2012-06-01 04:18:57.267954] I [client.c:2090:client_rpc_notify] 15-master-client-3: disconnected
>[2012-06-01 04:18:57.267972] E [afr-common.c:3664:afr_notify] 15-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:18:59.376049] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:00.139388] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:00.141377] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:00.141408] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:00.141602] I [afr-common.c:3627:afr_notify] 17-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:00.142281] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-0: Server lk version = 1
>[2012-06-01 04:19:00.146739] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:00.147422] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:00.147465] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:00.148623] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-1: Server lk version = 1
>[2012-06-01 04:19:00.149747] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:00.150910] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:00.150943] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:00.151122] I [afr-common.c:3627:afr_notify] 17-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:00.152608] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-2: Server lk version = 1
>[2012-06-01 04:19:00.165406] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:00.165902] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:00.166783] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:00.172534] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 17
>[2012-06-01 04:19:00.176010] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-3: Server lk version = 1
>[2012-06-01 04:19:00.178085] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 17-master-replicate-0: added root inode
>[2012-06-01 04:19:00.180182] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 17-master-replicate-1: added root inode
>[2012-06-01 04:19:00.218411] I [client.c:2151:notify] 16-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:00.218502] I [client.c:2151:notify] 16-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:00.218547] I [client.c:2151:notify] 16-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:00.218585] I [client.c:2151:notify] 16-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:00.219675] I [client.c:2090:client_rpc_notify] 16-master-client-0: disconnected
>[2012-06-01 04:19:00.219724] I [client.c:2090:client_rpc_notify] 16-master-client-1: disconnected
>[2012-06-01 04:19:00.219743] E [afr-common.c:3664:afr_notify] 16-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:00.220486] I [client.c:2090:client_rpc_notify] 16-master-client-2: disconnected
>[2012-06-01 04:19:00.221129] I [client.c:2090:client_rpc_notify] 16-master-client-3: disconnected
>[2012-06-01 04:19:00.221158] E [afr-common.c:3664:afr_notify] 16-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:00.449145] I [io-cache.c:1549:check_cache_size_ok] 18-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:00.449246] I [io-cache.c:1549:check_cache_size_ok] 18-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:00.456290] I [client.c:2142:notify] 18-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:00.461529] I [client.c:2142:notify] 18-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:00.465596] I [client.c:2142:notify] 18-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:00.470007] I [client.c:2142:notify] 18-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:00.479034] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:00.479161] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:00.479270] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:00.479375] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:04.166665] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:04.168191] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:04.168217] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:04.168392] I [afr-common.c:3627:afr_notify] 18-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:04.169370] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-0: Server lk version = 1
>[2012-06-01 04:19:04.175776] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:04.176525] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:04.176550] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:04.179206] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-1: Server lk version = 1
>[2012-06-01 04:19:04.179817] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:04.184553] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:04.184580] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:04.184661] I [afr-common.c:3627:afr_notify] 18-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:04.189917] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:04.190099] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-2: Server lk version = 1
>[2012-06-01 04:19:04.192043] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:04.192069] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:04.215492] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 18
>[2012-06-01 04:19:04.216604] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-3: Server lk version = 1
>[2012-06-01 04:19:04.219105] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 18-master-replicate-0: added root inode
>[2012-06-01 04:19:04.220899] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 18-master-replicate-1: added root inode
>[2012-06-01 04:19:04.226696] W [quota.c:2377:quota_fsync_cbk] 17-master-quota: quota context not set in inode (gfid:e12bbfb4-93a7-4301-a3ca-483cd705695b)
>[2012-06-01 04:19:04.244852] I [client.c:2151:notify] 17-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:04.244932] I [client.c:2151:notify] 17-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:04.244968] I [client.c:2151:notify] 17-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:04.245000] I [client.c:2151:notify] 17-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:04.247230] I [client.c:2090:client_rpc_notify] 17-master-client-0: disconnected
>[2012-06-01 04:19:04.247510] I [client.c:2090:client_rpc_notify] 17-master-client-1: disconnected
>[2012-06-01 04:19:04.247533] E [afr-common.c:3664:afr_notify] 17-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:04.247564] I [client.c:2090:client_rpc_notify] 17-master-client-2: disconnected
>[2012-06-01 04:19:04.247592] I [client.c:2090:client_rpc_notify] 17-master-client-3: disconnected
>[2012-06-01 04:19:04.247606] E [afr-common.c:3664:afr_notify] 17-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:07.670308] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:08.813295] I [io-cache.c:1549:check_cache_size_ok] 19-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:08.813366] I [io-cache.c:1549:check_cache_size_ok] 19-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:08.813417] I [quota.c:3085:quota_parse_limits] 19-master-quota: no "limit-set" option provided
>[2012-06-01 04:19:08.826596] I [client.c:2142:notify] 19-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:08.837253] I [client.c:2142:notify] 19-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:08.844252] I [client.c:2142:notify] 19-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:08.864081] I [client.c:2142:notify] 19-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:08.901363] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:08.901857] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:08.903025] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:08.903405] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:12.199545] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:12.200075] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:12.200469] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:12.200488] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:12.200606] I [afr-common.c:3627:afr_notify] 19-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:19:12.201004] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-1: Server lk version = 1
>[2012-06-01 04:19:12.201034] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:12.201046] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:12.202491] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-0: Server lk version = 1
>[2012-06-01 04:19:12.212521] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:12.212724] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:12.212961] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:12.212979] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:12.213087] I [afr-common.c:3627:afr_notify] 19-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:12.213120] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:12.213132] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:12.218925] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 19
>[2012-06-01 04:19:12.219721] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-2: Server lk version = 1
>[2012-06-01 04:19:12.219789] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-3: Server lk version = 1
>[2012-06-01 04:19:12.226089] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 19-master-replicate-0: added root inode
>[2012-06-01 04:19:12.229774] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 19-master-replicate-1: added root inode
>[2012-06-01 04:19:12.263507] I [client.c:2151:notify] 18-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:12.263572] I [client.c:2151:notify] 18-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:12.263601] I [client.c:2151:notify] 18-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:12.263626] I [client.c:2151:notify] 18-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:12.267390] I [client.c:2090:client_rpc_notify] 18-master-client-0: disconnected
>[2012-06-01 04:19:12.267439] I [client.c:2090:client_rpc_notify] 18-master-client-1: disconnected
>[2012-06-01 04:19:12.267457] E [afr-common.c:3664:afr_notify] 18-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:12.267492] I [client.c:2090:client_rpc_notify] 18-master-client-2: disconnected
>[2012-06-01 04:19:12.267545] I [client.c:2090:client_rpc_notify] 18-master-client-3: disconnected
>[2012-06-01 04:19:12.267567] E [afr-common.c:3664:afr_notify] 18-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:16.766570] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:17.798289] I [io-cache.c:1549:check_cache_size_ok] 20-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:17.798369] I [io-cache.c:1549:check_cache_size_ok] 20-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:17.814928] I [client.c:2142:notify] 20-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:17.822027] I [client.c:2142:notify] 20-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:17.827862] I [client.c:2142:notify] 20-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:17.831326] I [client.c:2142:notify] 20-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:17.854671] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:17.855532] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:17.856116] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:17.856199] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:20.048615] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:21.174145] I [io-cache.c:1549:check_cache_size_ok] 21-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:21.174217] I [io-cache.c:1549:check_cache_size_ok] 21-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:21.174285] I [quota.c:3085:quota_parse_limits] 21-master-quota: no "limit-set" option provided
>[2012-06-01 04:19:21.180129] I [client.c:2142:notify] 21-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:21.183032] I [client.c:2142:notify] 21-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:21.185665] I [client.c:2142:notify] 21-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:21.188345] I [client.c:2142:notify] 21-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:21.193685] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:21.193781] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:21.193858] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:21.193956] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:21.219525] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:21.220802] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:21.220829] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:21.220977] I [afr-common.c:3627:afr_notify] 20-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:21.221645] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-0: Server lk version = 1
>[2012-06-01 04:19:21.230589] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:21.232101] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:21.232123] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:21.232649] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-1: Server lk version = 1
>[2012-06-01 04:19:21.244516] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:21.245118] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:21.245139] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:21.245262] I [afr-common.c:3627:afr_notify] 20-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:21.245795] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-2: Server lk version = 1
>[2012-06-01 04:19:21.255011] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:21.255326] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:21.255350] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:21.258982] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 20
>[2012-06-01 04:19:21.260135] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-3: Server lk version = 1
>[2012-06-01 04:19:21.261173] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 20-master-replicate-0: added root inode
>[2012-06-01 04:19:21.262308] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 20-master-replicate-1: added root inode
>[2012-06-01 04:19:21.268855] W [quota.c:2377:quota_fsync_cbk] 19-master-quota: quota context not set in inode (gfid:fcb017d4-0e48-49a9-a2bc-def38d77c7ea)
>[2012-06-01 04:19:21.270969] I [client.c:2151:notify] 19-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:21.271028] I [client.c:2151:notify] 19-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:21.271069] I [client.c:2151:notify] 19-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:21.271096] I [client.c:2151:notify] 19-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:21.276730] I [client.c:2090:client_rpc_notify] 19-master-client-0: disconnected
>[2012-06-01 04:19:21.276777] I [client.c:2090:client_rpc_notify] 19-master-client-1: disconnected
>[2012-06-01 04:19:21.276790] E [afr-common.c:3664:afr_notify] 19-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:21.276815] I [client.c:2090:client_rpc_notify] 19-master-client-2: disconnected
>[2012-06-01 04:19:21.276841] I [client.c:2090:client_rpc_notify] 19-master-client-3: disconnected
>[2012-06-01 04:19:21.276852] E [afr-common.c:3664:afr_notify] 19-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:24.255041] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:24.255353] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:24.255374] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:24.255498] I [afr-common.c:3627:afr_notify] 21-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:24.257039] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-0: Server lk version = 1
>[2012-06-01 04:19:24.260899] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:24.261659] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:24.261681] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:24.262430] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-1: Server lk version = 1
>[2012-06-01 04:19:24.264827] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:24.265212] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:24.265234] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:24.265357] I [afr-common.c:3627:afr_notify] 21-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:24.266158] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-2: Server lk version = 1
>[2012-06-01 04:19:24.268690] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:24.270194] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:24.270217] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:24.274099] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 21
>[2012-06-01 04:19:24.274694] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-3: Server lk version = 1
>[2012-06-01 04:19:24.276432] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 21-master-replicate-0: added root inode
>[2012-06-01 04:19:24.277593] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 21-master-replicate-1: added root inode
>[2012-06-01 04:19:24.296487] I [client.c:2151:notify] 20-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:24.296555] I [client.c:2151:notify] 20-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:24.296592] I [client.c:2151:notify] 20-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:24.296644] I [client.c:2151:notify] 20-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:24.299744] I [client.c:2090:client_rpc_notify] 20-master-client-0: disconnected
>[2012-06-01 04:19:24.299783] I [client.c:2090:client_rpc_notify] 20-master-client-1: disconnected
>[2012-06-01 04:19:24.299798] E [afr-common.c:3664:afr_notify] 20-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:24.299822] I [client.c:2090:client_rpc_notify] 20-master-client-2: disconnected
>[2012-06-01 04:19:24.299848] I [client.c:2090:client_rpc_notify] 20-master-client-3: disconnected
>[2012-06-01 04:19:24.299860] E [afr-common.c:3664:afr_notify] 20-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:28.335322] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:29.414751] I [io-cache.c:1549:check_cache_size_ok] 22-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:29.414856] I [io-cache.c:1549:check_cache_size_ok] 22-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:29.424898] I [client.c:2142:notify] 22-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:29.434983] I [client.c:2142:notify] 22-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:29.444459] I [client.c:2142:notify] 22-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:29.454947] I [client.c:2142:notify] 22-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:29.473861] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:29.473984] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:29.474327] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:29.474456] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:33.278064] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:33.279304] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:33.279325] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:33.279532] I [afr-common.c:3627:afr_notify] 22-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:33.280724] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-0: Server lk version = 1
>[2012-06-01 04:19:33.280759] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:33.282178] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:33.282200] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:33.284825] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-1: Server lk version = 1
>[2012-06-01 04:19:33.291118] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:33.292811] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:33.292833] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:33.293074] I [afr-common.c:3627:afr_notify] 22-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:33.293695] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-2: Server lk version = 1
>[2012-06-01 04:19:33.310916] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:33.312269] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:33.312291] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:33.344677] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 22
>[2012-06-01 04:19:33.346129] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-3: Server lk version = 1
>[2012-06-01 04:19:33.352895] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 22-master-replicate-0: added root inode
>[2012-06-01 04:19:33.355217] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 22-master-replicate-1: added root inode
>[2012-06-01 04:19:33.359602] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:56b109e2-1571-42cb-aeb9-d38352a17da0)
>[2012-06-01 04:19:33.365200] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:8d1dde71-55d0-4ff8-a328-13e90fd16f92)
>[2012-06-01 04:19:33.370070] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:e9410e81-5eb7-4d96-9f9b-3a0126e5a10a)
>[2012-06-01 04:19:33.381895] I [client.c:2151:notify] 21-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:33.381963] I [client.c:2151:notify] 21-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:33.382005] I [client.c:2151:notify] 21-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:33.382035] I [client.c:2151:notify] 21-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:33.385182] I [client.c:2090:client_rpc_notify] 21-master-client-0: disconnected
>[2012-06-01 04:19:33.385215] I [client.c:2090:client_rpc_notify] 21-master-client-1: disconnected
>[2012-06-01 04:19:33.385227] E [afr-common.c:3664:afr_notify] 21-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:33.385249] I [client.c:2090:client_rpc_notify] 21-master-client-2: disconnected
>[2012-06-01 04:19:33.385272] I [client.c:2090:client_rpc_notify] 21-master-client-3: disconnected
>[2012-06-01 04:19:33.385282] E [afr-common.c:3664:afr_notify] 21-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:36.553629] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:37.724800] I [io-cache.c:1549:check_cache_size_ok] 23-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:37.724877] I [io-cache.c:1549:check_cache_size_ok] 23-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:37.724935] I [quota.c:3085:quota_parse_limits] 23-master-quota: no "limit-set" option provided
>[2012-06-01 04:19:37.740490] I [client.c:2142:notify] 23-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:37.743014] I [client.c:2142:notify] 23-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:37.748333] I [client.c:2142:notify] 23-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:37.755487] I [client.c:2142:notify] 23-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:37.765251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:37.765343] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:37.765546] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:37.765750] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:40.653436] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:41.323030] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:41.323206] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:41.323838] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:41.324292] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:41.324312] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:41.324427] I [afr-common.c:3627:afr_notify] 23-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:41.324464] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:41.324477] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:41.325107] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-0: Server lk version = 1
>[2012-06-01 04:19:41.325141] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:41.325155] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:41.325208] I [afr-common.c:3627:afr_notify] 23-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:41.325598] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-1: Server lk version = 1
>[2012-06-01 04:19:41.326252] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-2: Server lk version = 1
>[2012-06-01 04:19:41.334418] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:41.335826] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:41.335848] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:41.342254] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 23
>[2012-06-01 04:19:41.343730] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-3: Server lk version = 1
>[2012-06-01 04:19:41.346278] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 23-master-replicate-0: added root inode
>[2012-06-01 04:19:41.348809] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 23-master-replicate-1: added root inode
>[2012-06-01 04:19:41.364752] I [client.c:2151:notify] 22-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:41.364806] I [client.c:2151:notify] 22-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:41.364834] I [client.c:2151:notify] 22-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:41.364874] I [client.c:2151:notify] 22-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:41.368463] I [client.c:2090:client_rpc_notify] 22-master-client-0: disconnected
>[2012-06-01 04:19:41.368761] I [client.c:2090:client_rpc_notify] 22-master-client-1: disconnected
>[2012-06-01 04:19:41.368777] E [afr-common.c:3664:afr_notify] 22-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:41.368800] I [client.c:2090:client_rpc_notify] 22-master-client-2: disconnected
>[2012-06-01 04:19:41.368824] I [client.c:2090:client_rpc_notify] 22-master-client-3: disconnected
>[2012-06-01 04:19:41.368835] E [afr-common.c:3664:afr_notify] 22-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:41.735373] I [io-cache.c:1549:check_cache_size_ok] 24-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:41.735450] I [io-cache.c:1549:check_cache_size_ok] 24-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:41.741843] I [client.c:2142:notify] 24-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:41.744387] I [client.c:2142:notify] 24-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:41.746891] I [client.c:2142:notify] 24-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:41.749392] I [client.c:2142:notify] 24-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:41.754704] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:41.754803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:41.754897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:41.754972] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:44.093061] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:45.346800] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:45.348699] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:45.348732] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:45.352962] I [afr-common.c:3627:afr_notify] 24-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:45.357065] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-0: Server lk version = 1
>[2012-06-01 04:19:45.357111] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:45.359584] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:45.360372] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:45.361939] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-1: Server lk version = 1
>[2012-06-01 04:19:45.361998] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:45.363071] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:45.363104] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:45.363275] I [afr-common.c:3627:afr_notify] 24-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:45.363322] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:45.364433] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-2: Server lk version = 1
>[2012-06-01 04:19:45.364496] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:45.364519] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:45.372479] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 24
>[2012-06-01 04:19:45.373704] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-3: Server lk version = 1
>[2012-06-01 04:19:45.375412] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 24-master-replicate-0: added root inode
>[2012-06-01 04:19:45.377350] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 24-master-replicate-1: added root inode
>[2012-06-01 04:19:45.381595] W [quota.c:2377:quota_fsync_cbk] 23-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b)
>[2012-06-01 04:19:45.404453] I [io-cache.c:1549:check_cache_size_ok] 25-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:45.404578] I [io-cache.c:1549:check_cache_size_ok] 25-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:45.404698] I [quota.c:3085:quota_parse_limits] 25-master-quota: no "limit-set" option provided
>[2012-06-01 04:19:45.416794] I [client.c:2142:notify] 25-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:45.429920] I [client.c:2142:notify] 25-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:45.434302] I [client.c:2142:notify] 25-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:45.447348] I [client.c:2142:notify] 25-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:45.455002] W [quota.c:2377:quota_fsync_cbk] 23-master-quota: quota context not set in inode (gfid:d5f58101-467e-4b45-b5d4-518a2267eb5c)
>[2012-06-01 04:19:45.458045] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:45.459330] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:45.459451] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:45.459764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:45.478649] I [client.c:2151:notify] 23-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:45.478713] I [client.c:2151:notify] 23-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:45.478755] I [client.c:2151:notify] 23-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:45.478790] I [client.c:2151:notify] 23-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:45.484367] I [client.c:2090:client_rpc_notify] 23-master-client-0: disconnected
>[2012-06-01 04:19:45.484518] I [client.c:2090:client_rpc_notify] 23-master-client-1: disconnected
>[2012-06-01 04:19:45.484539] E [afr-common.c:3664:afr_notify] 23-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:45.484571] I [client.c:2090:client_rpc_notify] 23-master-client-2: disconnected
>[2012-06-01 04:19:45.484603] I [client.c:2090:client_rpc_notify] 23-master-client-3: disconnected
>[2012-06-01 04:19:45.484619] E [afr-common.c:3664:afr_notify] 23-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:47.733563] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:48.798395] I [io-cache.c:1549:check_cache_size_ok] 26-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:48.798470] I [io-cache.c:1549:check_cache_size_ok] 26-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:48.808455] I [client.c:2142:notify] 26-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:48.811031] I [client.c:2142:notify] 26-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:48.813580] I [client.c:2142:notify] 26-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:48.816173] I [client.c:2142:notify] 26-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:48.827825] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:48.827897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:48.827950] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:48.828002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:49.371668] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:49.380227] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:49.380274] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:49.381280] I [afr-common.c:3627:afr_notify] 25-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:49.382321] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-0: Server lk version = 1
>[2012-06-01 04:19:49.384776] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:49.389656] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:49.389691] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:49.390333] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-1: Server lk version = 1
>[2012-06-01 04:19:49.390696] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:49.397456] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:49.397490] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:49.397726] I [afr-common.c:3627:afr_notify] 25-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:49.398781] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-2: Server lk version = 1
>[2012-06-01 04:19:49.407132] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:49.407892] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:49.407924] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:49.635373] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 25
>[2012-06-01 04:19:49.637024] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-3: Server lk version = 1
>[2012-06-01 04:19:49.638278] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 25-master-replicate-0: added root inode
>[2012-06-01 04:19:49.639413] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 25-master-replicate-1: added root inode
>[2012-06-01 04:19:49.666816] I [client.c:2151:notify] 24-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:49.666879] I [client.c:2151:notify] 24-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:49.666909] I [client.c:2151:notify] 24-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:49.666935] I [client.c:2151:notify] 24-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:49.670369] I [client.c:2090:client_rpc_notify] 24-master-client-0: disconnected
>[2012-06-01 04:19:49.670406] I [client.c:2090:client_rpc_notify] 24-master-client-1: disconnected
>[2012-06-01 04:19:49.670419] E [afr-common.c:3664:afr_notify] 24-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:49.670445] I [client.c:2090:client_rpc_notify] 24-master-client-2: disconnected
>[2012-06-01 04:19:49.670469] I [client.c:2090:client_rpc_notify] 24-master-client-3: disconnected
>[2012-06-01 04:19:49.670480] E [afr-common.c:3664:afr_notify] 24-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:51.281431] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:52.408662] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:52.409003] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:52.409030] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:52.409448] I [afr-common.c:3627:afr_notify] 26-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:19:52.409527] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-1: Server lk version = 1
>[2012-06-01 04:19:52.409586] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:52.409970] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:52.410710] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:52.411297] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-0: Server lk version = 1
>[2012-06-01 04:19:52.430716] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:52.431094] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:52.431113] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:52.431227] I [afr-common.c:3627:afr_notify] 26-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:52.432345] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-2: Server lk version = 1
>[2012-06-01 04:19:52.452085] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:52.452524] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:52.452544] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:52.469816] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 26
>[2012-06-01 04:19:52.470980] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-3: Server lk version = 1
>[2012-06-01 04:19:52.472581] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 26-master-replicate-0: added root inode
>[2012-06-01 04:19:52.474320] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 26-master-replicate-1: added root inode
>[2012-06-01 04:19:52.521065] W [quota.c:2377:quota_fsync_cbk] 25-master-quota: quota context not set in inode (gfid:3f282348-54c0-4e62-83c9-a6d9e4bd91f6)
>[2012-06-01 04:19:52.526374] W [quota.c:2377:quota_fsync_cbk] 25-master-quota: quota context not set in inode (gfid:978564f8-aa12-48bd-9979-9ddd41eec618)
>[2012-06-01 04:19:52.530661] I [client.c:2151:notify] 25-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:52.530713] I [client.c:2151:notify] 25-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:52.530750] I [client.c:2151:notify] 25-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:52.530782] I [client.c:2151:notify] 25-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:52.533639] I [client.c:2090:client_rpc_notify] 25-master-client-0: disconnected
>[2012-06-01 04:19:52.533676] I [client.c:2090:client_rpc_notify] 25-master-client-1: disconnected
>[2012-06-01 04:19:52.533689] E [afr-common.c:3664:afr_notify] 25-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:52.533718] I [client.c:2090:client_rpc_notify] 25-master-client-2: disconnected
>[2012-06-01 04:19:52.533743] I [client.c:2090:client_rpc_notify] 25-master-client-3: disconnected
>[2012-06-01 04:19:52.533752] E [afr-common.c:3664:afr_notify] 25-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:52.554749] I [io-cache.c:1549:check_cache_size_ok] 27-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:52.554847] I [io-cache.c:1549:check_cache_size_ok] 27-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:52.562897] I [quota.c:3085:quota_parse_limits] 27-master-quota: no "limit-set" option provided
>[2012-06-01 04:19:52.674783] I [client.c:2142:notify] 27-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:52.681630] I [client.c:2142:notify] 27-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:52.707695] I [client.c:2142:notify] 27-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:52.712495] I [client.c:2142:notify] 27-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:52.751606] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:52.751721] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:52.751800] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:52.752306] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:55.088789] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:56.164014] I [io-cache.c:1549:check_cache_size_ok] 28-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:19:56.164091] I [io-cache.c:1549:check_cache_size_ok] 28-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:19:56.171217] I [client.c:2142:notify] 28-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:56.179404] I [client.c:2142:notify] 28-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:56.184926] I [client.c:2142:notify] 28-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:19:56.191057] I [client.c:2142:notify] 28-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:19:56.198308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:19:56.198429] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:19:56.199066] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:19:56.199148] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:19:56.457406] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:56.458216] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:56.458241] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:56.458358] I [afr-common.c:3627:afr_notify] 27-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:56.458666] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-0: Server lk version = 1
>[2012-06-01 04:19:56.459267] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:56.459861] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:56.460710] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:56.461034] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-1: Server lk version = 1
>[2012-06-01 04:19:56.471593] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:56.473692] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:56.473713] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:56.473836] I [afr-common.c:3627:afr_notify] 27-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:56.474642] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-2: Server lk version = 1
>[2012-06-01 04:19:56.481300] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:56.482943] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:56.482962] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:56.487666] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 27
>[2012-06-01 04:19:56.488692] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-3: Server lk version = 1
>[2012-06-01 04:19:56.489310] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 27-master-replicate-0: added root inode
>[2012-06-01 04:19:56.490572] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 27-master-replicate-1: added root inode
>[2012-06-01 04:19:56.499652] I [client.c:2151:notify] 26-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:56.499713] I [client.c:2151:notify] 26-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:56.499743] I [client.c:2151:notify] 26-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:56.499778] I [client.c:2151:notify] 26-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:56.502314] I [client.c:2090:client_rpc_notify] 26-master-client-0: disconnected
>[2012-06-01 04:19:56.502351] I [client.c:2090:client_rpc_notify] 26-master-client-1: disconnected
>[2012-06-01 04:19:56.502363] E [afr-common.c:3664:afr_notify] 26-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:56.502410] I [client.c:2090:client_rpc_notify] 26-master-client-2: disconnected
>[2012-06-01 04:19:56.502434] I [client.c:2090:client_rpc_notify] 26-master-client-3: disconnected
>[2012-06-01 04:19:56.502445] E [afr-common.c:3664:afr_notify] 26-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:58.491766] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:19:59.490954] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:59.491488] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:59.492527] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:19:59.492550] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:59.492693] I [afr-common.c:3627:afr_notify] 28-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:19:59.493874] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:19:59.493896] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:59.498576] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-0: Server lk version = 1
>[2012-06-01 04:19:59.498608] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-1: Server lk version = 1
>[2012-06-01 04:19:59.502361] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:59.504109] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:19:59.504653] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:59.504780] I [afr-common.c:3627:afr_notify] 28-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:19:59.506241] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-2: Server lk version = 1
>[2012-06-01 04:19:59.511808] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:19:59.513912] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:19:59.514546] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:19:59.542017] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 28
>[2012-06-01 04:19:59.544814] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-3: Server lk version = 1
>[2012-06-01 04:19:59.555984] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 28-master-replicate-0: added root inode
>[2012-06-01 04:19:59.561016] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 28-master-replicate-1: added root inode
>[2012-06-01 04:19:59.584015] W [quota.c:2377:quota_fsync_cbk] 27-master-quota: quota context not set in inode (gfid:8908ad73-a402-4e97-afb5-b551b7fe9359)
>[2012-06-01 04:19:59.586125] I [client.c:2151:notify] 27-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:59.586187] I [client.c:2151:notify] 27-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:59.586222] I [client.c:2151:notify] 27-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:59.586251] I [client.c:2151:notify] 27-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:19:59.588353] I [client.c:2090:client_rpc_notify] 27-master-client-0: disconnected
>[2012-06-01 04:19:59.588409] I [client.c:2090:client_rpc_notify] 27-master-client-1: disconnected
>[2012-06-01 04:19:59.588424] E [afr-common.c:3664:afr_notify] 27-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:19:59.588448] I [client.c:2090:client_rpc_notify] 27-master-client-2: disconnected
>[2012-06-01 04:19:59.588471] I [client.c:2090:client_rpc_notify] 27-master-client-3: disconnected
>[2012-06-01 04:19:59.588483] E [afr-common.c:3664:afr_notify] 27-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:09.812466] I [io-cache.c:1549:check_cache_size_ok] 29-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:09.812651] I [io-cache.c:1549:check_cache_size_ok] 29-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:09.812714] I [quota.c:3085:quota_parse_limits] 29-master-quota: no "limit-set" option provided
>[2012-06-01 04:20:09.819834] I [client.c:2142:notify] 29-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:09.824101] I [client.c:2142:notify] 29-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:09.826971] I [client.c:2142:notify] 29-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:09.829552] I [client.c:2142:notify] 29-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:09.835705] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:09.835812] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:09.835894] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:09.835975] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:13.003075] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:13.536674] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:13.538057] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:13.538129] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:13.538464] I [afr-common.c:3627:afr_notify] 29-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:13.543082] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-0: Server lk version = 1
>[2012-06-01 04:20:13.545925] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:13.546814] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:13.546844] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:13.548857] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-1: Server lk version = 1
>[2012-06-01 04:20:13.561265] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:13.561912] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:13.562158] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:13.562338] I [afr-common.c:3627:afr_notify] 29-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:13.562768] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-2: Server lk version = 1
>[2012-06-01 04:20:13.567729] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:13.569157] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:13.569198] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:13.574083] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 29
>[2012-06-01 04:20:13.574434] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-3: Server lk version = 1
>[2012-06-01 04:20:13.581574] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 29-master-replicate-0: added root inode
>[2012-06-01 04:20:13.584979] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 29-master-replicate-1: added root inode
>[2012-06-01 04:20:13.634135] I [client.c:2151:notify] 28-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:13.634200] I [client.c:2151:notify] 28-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:13.634261] I [client.c:2151:notify] 28-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:13.634304] I [client.c:2151:notify] 28-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:13.641185] I [client.c:2090:client_rpc_notify] 28-master-client-0: disconnected
>[2012-06-01 04:20:13.641238] I [client.c:2090:client_rpc_notify] 28-master-client-1: disconnected
>[2012-06-01 04:20:13.641258] E [afr-common.c:3664:afr_notify] 28-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:13.641291] I [client.c:2090:client_rpc_notify] 28-master-client-2: disconnected
>[2012-06-01 04:20:13.641325] I [client.c:2090:client_rpc_notify] 28-master-client-3: disconnected
>[2012-06-01 04:20:13.641342] E [afr-common.c:3664:afr_notify] 28-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:14.094922] I [io-cache.c:1549:check_cache_size_ok] 30-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:14.095065] I [io-cache.c:1549:check_cache_size_ok] 30-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:14.108310] I [client.c:2142:notify] 30-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:14.121884] I [client.c:2142:notify] 30-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:14.126324] I [client.c:2142:notify] 30-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:14.136263] I [client.c:2142:notify] 30-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:14.159287] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:14.160494] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:14.160708] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:14.160833] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:16.224023] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:17.396062] I [io-cache.c:1549:check_cache_size_ok] 31-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:17.396142] I [io-cache.c:1549:check_cache_size_ok] 31-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:17.396206] I [quota.c:3085:quota_parse_limits] 31-master-quota: no "limit-set" option provided
>[2012-06-01 04:20:17.401562] I [client.c:2142:notify] 31-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:17.416095] I [client.c:2142:notify] 31-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:17.421219] I [client.c:2142:notify] 31-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:17.429778] I [client.c:2142:notify] 31-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:17.444496] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:17.444743] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:17.444818] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:17.444982] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:17.571642] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:17.572361] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:17.572396] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:17.572661] I [afr-common.c:3627:afr_notify] 30-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:17.575177] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-0: Server lk version = 1
>[2012-06-01 04:20:17.575208] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:17.576218] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:17.576284] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:17.578979] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-1: Server lk version = 1
>[2012-06-01 04:20:17.588895] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:17.589836] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:17.589857] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:17.590417] I [afr-common.c:3627:afr_notify] 30-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:17.596928] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-2: Server lk version = 1
>[2012-06-01 04:20:17.599985] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:17.600530] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:17.600551] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:17.608026] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 30
>[2012-06-01 04:20:17.608504] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-3: Server lk version = 1
>[2012-06-01 04:20:17.616461] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 30-master-replicate-0: added root inode
>[2012-06-01 04:20:17.618944] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 30-master-replicate-1: added root inode
>[2012-06-01 04:20:17.628809] W [quota.c:2377:quota_fsync_cbk] 29-master-quota: quota context not set in inode (gfid:b94ab6ab-6467-42b7-a6d4-e90673e21dbc)
>[2012-06-01 04:20:17.652844] I [client.c:2151:notify] 29-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:17.652903] I [client.c:2151:notify] 29-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:17.652961] I [client.c:2151:notify] 29-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:17.653031] I [client.c:2151:notify] 29-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:17.655950] I [socket.c:2315:socket_submit_request] 29-master-client-1: not connected (priv->connected = 255)
>[2012-06-01 04:20:17.655970] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-1: failed to submit rpc-request (XID: 0x3889x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-1)
>[2012-06-01 04:20:17.655997] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:17.656024] I [socket.c:2315:socket_submit_request] 29-master-client-0: not connected (priv->connected = 255)
>[2012-06-01 04:20:17.656065] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-0: failed to submit rpc-request (XID: 0x3885x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-0)
>[2012-06-01 04:20:17.656095] W [client3_1-fops.c:881:client3_1_flush_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:17.656166] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-1: failed to submit rpc-request (XID: 0x3890x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-1)
>[2012-06-01 04:20:17.656183] W [client3_1-fops.c:881:client3_1_flush_cbk] 29-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:17.656230] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-0: failed to submit rpc-request (XID: 0x3886x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0)
>[2012-06-01 04:20:17.656269] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:17.656315] I [socket.c:1798:socket_event_handler] 29-transport: disconnecting now
>[2012-06-01 04:20:17.657770] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 29-master-client-0: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:20:17.651088 (xid=0x3884x)
>[2012-06-01 04:20:17.657801] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:17.657815] I [client.c:2090:client_rpc_notify] 29-master-client-0: disconnected
>[2012-06-01 04:20:17.657854] I [socket.c:1798:socket_event_handler] 29-transport: disconnecting now
>[2012-06-01 04:20:17.657879] I [client.c:2090:client_rpc_notify] 29-master-client-1: disconnected
>[2012-06-01 04:20:17.657891] E [afr-common.c:3664:afr_notify] 29-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:17.658018] I [client.c:2090:client_rpc_notify] 29-master-client-2: disconnected
>[2012-06-01 04:20:17.658078] I [client.c:2090:client_rpc_notify] 29-master-client-3: disconnected
>[2012-06-01 04:20:17.658114] E [afr-common.c:3664:afr_notify] 29-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:20.617145] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:20.617444] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:20.618261] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:20.618294] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:20.618496] I [afr-common.c:3627:afr_notify] 31-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:20.618552] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:20.620294] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:20.621275] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-0: Server lk version = 1
>[2012-06-01 04:20:20.621319] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-1: Server lk version = 1
>[2012-06-01 04:20:20.642224] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:20.648230] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:20.648265] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:20.648469] I [afr-common.c:3627:afr_notify] 31-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:20.649275] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-2: Server lk version = 1
>[2012-06-01 04:20:20.654315] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:20.657040] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:20.657076] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:20.696553] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 31
>[2012-06-01 04:20:20.698301] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-3: Server lk version = 1
>[2012-06-01 04:20:20.704475] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 31-master-replicate-0: added root inode
>[2012-06-01 04:20:20.711682] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 31-master-replicate-1: added root inode
>[2012-06-01 04:20:20.751317] I [client.c:2151:notify] 30-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:20.751402] I [client.c:2151:notify] 30-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:20.751447] I [client.c:2151:notify] 30-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:20.751485] I [client.c:2151:notify] 30-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:20.757687] I [client.c:2090:client_rpc_notify] 30-master-client-0: disconnected
>[2012-06-01 04:20:20.757738] I [client.c:2090:client_rpc_notify] 30-master-client-1: disconnected
>[2012-06-01 04:20:20.757758] E [afr-common.c:3664:afr_notify] 30-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:20.757790] I [client.c:2090:client_rpc_notify] 30-master-client-2: disconnected
>[2012-06-01 04:20:20.757824] I [client.c:2090:client_rpc_notify] 30-master-client-3: disconnected
>[2012-06-01 04:20:20.757841] E [afr-common.c:3664:afr_notify] 30-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:24.791568] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:25.874817] I [io-cache.c:1549:check_cache_size_ok] 32-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:25.874912] I [io-cache.c:1549:check_cache_size_ok] 32-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:25.881543] I [client.c:2142:notify] 32-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:25.884608] I [client.c:2142:notify] 32-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:25.887143] I [client.c:2142:notify] 32-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:25.889552] I [client.c:2142:notify] 32-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:25.895519] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:25.895619] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:25.895722] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:25.895848] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:28.459730] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:29.660129] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:29.660925] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:29.660946] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:29.661062] I [afr-common.c:3627:afr_notify] 32-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:29.662099] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-0: Server lk version = 1
>[2012-06-01 04:20:29.666368] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:29.667057] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:29.667077] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:29.668714] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-1: Server lk version = 1
>[2012-06-01 04:20:29.677104] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:29.678832] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:29.679035] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:29.679055] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:29.679173] I [afr-common.c:3627:afr_notify] 32-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:29.679778] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:29.679798] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:29.687844] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 32
>[2012-06-01 04:20:29.687925] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-2: Server lk version = 1
>[2012-06-01 04:20:29.688796] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-3: Server lk version = 1
>[2012-06-01 04:20:29.693925] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 32-master-replicate-0: added root inode
>[2012-06-01 04:20:29.696109] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 32-master-replicate-1: added root inode
>[2012-06-01 04:20:29.710238] W [quota.c:2377:quota_fsync_cbk] 31-master-quota: quota context not set in inode (gfid:37ca1a7f-d644-44e9-b0bf-1a62d7380159)
>[2012-06-01 04:20:29.712637] I [client.c:2151:notify] 31-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:29.712698] I [client.c:2151:notify] 31-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:29.712729] I [client.c:2151:notify] 31-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:29.712756] I [client.c:2151:notify] 31-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:29.715924] I [client.c:2090:client_rpc_notify] 31-master-client-0: disconnected
>[2012-06-01 04:20:29.715962] I [client.c:2090:client_rpc_notify] 31-master-client-1: disconnected
>[2012-06-01 04:20:29.715979] E [afr-common.c:3664:afr_notify] 31-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:29.716002] I [client.c:2090:client_rpc_notify] 31-master-client-2: disconnected
>[2012-06-01 04:20:29.716024] I [client.c:2090:client_rpc_notify] 31-master-client-3: disconnected
>[2012-06-01 04:20:29.716035] E [afr-common.c:3664:afr_notify] 31-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:39.555504] I [io-cache.c:1549:check_cache_size_ok] 33-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:39.555632] I [io-cache.c:1549:check_cache_size_ok] 33-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:39.555745] I [quota.c:3085:quota_parse_limits] 33-master-quota: no "limit-set" option provided
>[2012-06-01 04:20:39.563009] I [client.c:2142:notify] 33-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:39.568233] I [client.c:2142:notify] 33-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:39.573438] I [client.c:2142:notify] 33-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:39.579580] I [client.c:2142:notify] 33-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:39.593153] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:39.593308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:39.593490] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:39.593619] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:41.924164] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:42.701091] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:42.701359] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:42.702956] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:42.702987] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:42.703177] I [afr-common.c:3627:afr_notify] 33-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:42.703229] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:42.703249] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:42.704569] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-0: Server lk version = 1
>[2012-06-01 04:20:42.704639] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-1: Server lk version = 1
>[2012-06-01 04:20:42.706303] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:42.708765] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:42.708796] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:42.708897] I [afr-common.c:3627:afr_notify] 33-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:42.711522] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-2: Server lk version = 1
>[2012-06-01 04:20:42.717065] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:42.717699] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:42.717729] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:42.723486] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 33
>[2012-06-01 04:20:42.725698] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-3: Server lk version = 1
>[2012-06-01 04:20:42.727237] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 33-master-replicate-0: added root inode
>[2012-06-01 04:20:42.728304] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 33-master-replicate-1: added root inode
>[2012-06-01 04:20:42.733049] I [fuse-bridge.c:3572:fuse_migrate_fd] 0-glusterfs-fuse: create call on fd (0xf784b14) is in progress, hence waiting
>[2012-06-01 04:20:42.963258] I [io-cache.c:1549:check_cache_size_ok] 34-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:42.963361] I [io-cache.c:1549:check_cache_size_ok] 34-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:42.978316] I [client.c:2142:notify] 34-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:42.984907] I [client.c:2142:notify] 34-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:42.999615] I [client.c:2142:notify] 34-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:43.004804] I [client.c:2142:notify] 34-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:43.020345] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:43.020457] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:43.020565] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:43.020660] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:43.743132] I [client.c:2151:notify] 32-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:43.743227] I [client.c:2151:notify] 32-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:43.743273] I [client.c:2151:notify] 32-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:43.743336] I [client.c:2151:notify] 32-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:43.745709] I [client.c:2090:client_rpc_notify] 32-master-client-0: disconnected
>[2012-06-01 04:20:43.745763] I [client.c:2090:client_rpc_notify] 32-master-client-1: disconnected
>[2012-06-01 04:20:43.745784] E [afr-common.c:3664:afr_notify] 32-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:43.745819] I [client.c:2090:client_rpc_notify] 32-master-client-2: disconnected
>[2012-06-01 04:20:43.745855] I [client.c:2090:client_rpc_notify] 32-master-client-3: disconnected
>[2012-06-01 04:20:43.745907] E [afr-common.c:3664:afr_notify] 32-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:45.791230] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:20:46.717734] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:46.719451] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:46.719545] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:46.719750] I [afr-common.c:3627:afr_notify] 34-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:46.720904] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-0: Server lk version = 1
>[2012-06-01 04:20:46.726816] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:46.727936] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:46.727957] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:46.729265] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-1: Server lk version = 1
>[2012-06-01 04:20:46.734516] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:46.736067] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:46.736091] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:46.736367] I [afr-common.c:3627:afr_notify] 34-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:46.736656] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-2: Server lk version = 1
>[2012-06-01 04:20:46.737062] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:46.737317] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:46.737395] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:46.742173] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 34
>[2012-06-01 04:20:46.743061] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-3: Server lk version = 1
>[2012-06-01 04:20:46.743887] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 34-master-replicate-0: added root inode
>[2012-06-01 04:20:46.745001] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 34-master-replicate-1: added root inode
>[2012-06-01 04:20:46.748329] I [client.c:2151:notify] 33-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:46.748591] I [client.c:2151:notify] 33-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:46.748993] I [client.c:2151:notify] 33-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:46.749505] I [client.c:2151:notify] 33-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:46.750656] I [socket.c:1798:socket_event_handler] 33-transport: disconnecting now
>[2012-06-01 04:20:46.750735] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 33-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:20:46.748245 (xid=0x1955x)
>[2012-06-01 04:20:46.750757] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.750778] I [socket.c:2315:socket_submit_request] 33-master-client-0: not connected (priv->connected = 255)
>[2012-06-01 04:20:46.750791] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1957x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-0)
>[2012-06-01 04:20:46.750802] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.755106] I [socket.c:2315:socket_submit_request] 33-master-client-1: not connected (priv->connected = 0)
>[2012-06-01 04:20:46.755131] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-1: failed to submit rpc-request (XID: 0x1956x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-1)
>[2012-06-01 04:20:46.755144] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.755166] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1958x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0)
>[2012-06-01 04:20:46.755186] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.755711] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 33-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2012-06-01 04:20:46.748166 (xid=0x1954x)
>[2012-06-01 04:20:46.755732] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.755745] I [client.c:2090:client_rpc_notify] 33-master-client-1: disconnected
>[2012-06-01 04:20:46.756051] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1959x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0)
>[2012-06-01 04:20:46.756073] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:20:46.756087] W [client3_1-fops.c:5267:client3_1_finodelk] 33-master-client-1:  (e60426bf-3996-4407-9342-4306d3d71de0) remote_fd is -1. EBADFD
>[2012-06-01 04:20:46.756107] I [socket.c:1798:socket_event_handler] 33-transport: disconnecting now
>[2012-06-01 04:20:46.756132] I [client.c:2090:client_rpc_notify] 33-master-client-0: disconnected
>[2012-06-01 04:20:46.756144] E [afr-common.c:3664:afr_notify] 33-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:46.756316] I [client.c:2090:client_rpc_notify] 33-master-client-2: disconnected
>[2012-06-01 04:20:46.756350] I [client.c:2090:client_rpc_notify] 33-master-client-3: disconnected
>[2012-06-01 04:20:46.756362] E [afr-common.c:3664:afr_notify] 33-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:46.758797] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:46.759643] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:46.759664] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:46.759673] I [client-handshake.c:1282:client_post_handshake] 33-master-client-1: 1 fds open - Delaying child_up until they are re-opened
>[2012-06-01 04:20:46.762023] I [client-lk.c:601:decrement_reopen_fd_count] 33-master-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>[2012-06-01 04:20:46.762110] I [afr-common.c:3627:afr_notify] 33-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:20:46.764159] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-1: Server lk version = 1
>[2012-06-01 04:20:46.953794] I [io-cache.c:1549:check_cache_size_ok] 35-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:20:46.953872] I [io-cache.c:1549:check_cache_size_ok] 35-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:20:46.953944] I [quota.c:3085:quota_parse_limits] 35-master-quota: no "limit-set" option provided
>[2012-06-01 04:20:46.961880] I [client.c:2142:notify] 35-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:46.966151] I [client.c:2142:notify] 35-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:46.968863] I [client.c:2142:notify] 35-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:20:46.971438] I [client.c:2142:notify] 35-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:20:46.981157] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:20:46.981254] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:20:46.981337] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:20:46.981411] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:20:50.747530] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:50.747944] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:20:50.747967] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:50.748086] I [afr-common.c:3627:afr_notify] 35-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:20:50.750859] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-0: Server lk version = 1
>[2012-06-01 04:20:50.755708] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:50.756118] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:20:50.756139] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:50.759015] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-1: Server lk version = 1
>[2012-06-01 04:20:50.760056] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:50.762671] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:20:50.762692] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:50.763497] I [afr-common.c:3627:afr_notify] 35-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:20:50.765084] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-2: Server lk version = 1
>[2012-06-01 04:20:50.768128] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:20:50.768709] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:20:50.768730] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:20:50.798994] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 35
>[2012-06-01 04:20:50.799481] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-3: Server lk version = 1
>[2012-06-01 04:20:50.800424] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 35-master-replicate-0: added root inode
>[2012-06-01 04:20:50.803270] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 35-master-replicate-1: added root inode
>[2012-06-01 04:20:50.816878] I [client.c:2151:notify] 34-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:50.816927] I [client.c:2151:notify] 34-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:50.816954] I [client.c:2151:notify] 34-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:50.816982] I [client.c:2151:notify] 34-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:20:50.821042] I [client.c:2090:client_rpc_notify] 34-master-client-0: disconnected
>[2012-06-01 04:20:50.821080] I [client.c:2090:client_rpc_notify] 34-master-client-1: disconnected
>[2012-06-01 04:20:50.821093] E [afr-common.c:3664:afr_notify] 34-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:50.821118] I [client.c:2090:client_rpc_notify] 34-master-client-2: disconnected
>[2012-06-01 04:20:50.821143] I [client.c:2090:client_rpc_notify] 34-master-client-3: disconnected
>[2012-06-01 04:20:50.821156] E [afr-common.c:3664:afr_notify] 34-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:20:59.493594] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:00.580731] I [io-cache.c:1549:check_cache_size_ok] 36-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:00.580808] I [io-cache.c:1549:check_cache_size_ok] 36-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:00.642068] I [client.c:2142:notify] 36-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:00.651639] I [client.c:2142:notify] 36-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:00.654683] I [client.c:2142:notify] 36-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:00.665665] I [client.c:2142:notify] 36-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:00.675506] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:00.676165] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:00.676251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:00.676402] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:03.107194] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:03.783749] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:03.783827] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:03.784558] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:03.785390] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:03.785410] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:03.785521] I [afr-common.c:3627:afr_notify] 36-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:03.785564] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:03.785578] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:03.785958] I [afr-common.c:3627:afr_notify] 36-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:03.785995] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:03.786008] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:03.786984] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-2: Server lk version = 1
>[2012-06-01 04:21:03.787018] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-0: Server lk version = 1
>[2012-06-01 04:21:03.787047] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-1: Server lk version = 1
>[2012-06-01 04:21:03.795608] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:03.795896] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:03.795915] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:03.801552] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 36
>[2012-06-01 04:21:03.802327] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-3: Server lk version = 1
>[2012-06-01 04:21:03.804721] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 36-master-replicate-0: added root inode
>[2012-06-01 04:21:03.806026] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 36-master-replicate-1: added root inode
>[2012-06-01 04:21:03.824086] I [client.c:2151:notify] 35-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:03.824146] I [client.c:2151:notify] 35-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:03.824174] I [client.c:2151:notify] 35-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:03.824197] I [client.c:2151:notify] 35-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:03.827130] I [client.c:2090:client_rpc_notify] 35-master-client-0: disconnected
>[2012-06-01 04:21:03.827165] I [client.c:2090:client_rpc_notify] 35-master-client-1: disconnected
>[2012-06-01 04:21:03.827177] E [afr-common.c:3664:afr_notify] 35-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:03.827413] I [client.c:2090:client_rpc_notify] 35-master-client-2: disconnected
>[2012-06-01 04:21:03.827442] I [client.c:2090:client_rpc_notify] 35-master-client-3: disconnected
>[2012-06-01 04:21:03.827453] E [afr-common.c:3664:afr_notify] 35-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:04.220817] I [io-cache.c:1549:check_cache_size_ok] 37-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:04.220897] I [io-cache.c:1549:check_cache_size_ok] 37-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:04.220960] I [quota.c:3085:quota_parse_limits] 37-master-quota: no "limit-set" option provided
>[2012-06-01 04:21:04.318148] I [client.c:2142:notify] 37-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:04.326512] I [client.c:2142:notify] 37-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:04.329949] I [client.c:2142:notify] 37-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:04.333644] I [client.c:2142:notify] 37-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:04.340085] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:04.340179] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:04.340257] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:04.340333] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:06.667242] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:07.747770] I [io-cache.c:1549:check_cache_size_ok] 38-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:07.747842] I [io-cache.c:1549:check_cache_size_ok] 38-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:07.759300] I [client.c:2142:notify] 38-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:07.768099] I [client.c:2142:notify] 38-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:07.770763] I [client.c:2142:notify] 38-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:07.773318] I [client.c:2142:notify] 38-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:07.787644] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:07.787731] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:07.788225] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:07.788310] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:07.798748] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:07.800684] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:07.800817] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:07.801379] I [afr-common.c:3627:afr_notify] 37-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:07.801447] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-0: Server lk version = 1
>[2012-06-01 04:21:07.809572] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:07.810053] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:07.810072] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:07.813893] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-1: Server lk version = 1
>[2012-06-01 04:21:07.813922] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:07.814299] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:07.814317] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:07.814421] I [afr-common.c:3627:afr_notify] 37-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:07.815254] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-2: Server lk version = 1
>[2012-06-01 04:21:07.815922] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:07.817065] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:07.817085] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:07.835233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 37
>[2012-06-01 04:21:07.837045] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-3: Server lk version = 1
>[2012-06-01 04:21:07.840303] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 37-master-replicate-0: added root inode
>[2012-06-01 04:21:07.843308] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 37-master-replicate-1: added root inode
>[2012-06-01 04:21:07.868585] I [client.c:2151:notify] 36-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:07.868670] I [client.c:2151:notify] 36-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:07.868700] I [client.c:2151:notify] 36-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:07.868735] I [client.c:2151:notify] 36-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:07.872537] I [client.c:2090:client_rpc_notify] 36-master-client-0: disconnected
>[2012-06-01 04:21:07.872573] I [client.c:2090:client_rpc_notify] 36-master-client-1: disconnected
>[2012-06-01 04:21:07.872586] E [afr-common.c:3664:afr_notify] 36-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:07.872638] I [client.c:2090:client_rpc_notify] 36-master-client-2: disconnected
>[2012-06-01 04:21:07.872665] I [client.c:2090:client_rpc_notify] 36-master-client-3: disconnected
>[2012-06-01 04:21:07.872676] E [afr-common.c:3664:afr_notify] 36-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:10.050508] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:10.820528] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:10.823160] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:10.823185] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:10.823228] I [afr-common.c:3627:afr_notify] 38-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:10.826341] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-0: Server lk version = 1
>[2012-06-01 04:21:10.826376] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:10.827248] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:10.827269] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:10.830696] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-1: Server lk version = 1
>[2012-06-01 04:21:10.832432] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:10.832662] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:10.833112] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:10.833243] I [afr-common.c:3627:afr_notify] 38-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:21:10.834881] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-3: Server lk version = 1
>[2012-06-01 04:21:10.835704] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:10.836788] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:10.836826] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:10.841057] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 38
>[2012-06-01 04:21:10.841855] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-2: Server lk version = 1
>[2012-06-01 04:21:10.844659] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 38-master-replicate-0: added root inode
>[2012-06-01 04:21:10.846483] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 38-master-replicate-1: added root inode
>[2012-06-01 04:21:10.865280] W [quota.c:2377:quota_fsync_cbk] 37-master-quota: quota context not set in inode (gfid:c47d718d-1349-420e-83a3-2d6a59c22f7b)
>[2012-06-01 04:21:10.867579] I [client.c:2151:notify] 37-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:10.867647] I [client.c:2151:notify] 37-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:10.867686] I [client.c:2151:notify] 37-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:10.867713] I [client.c:2151:notify] 37-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:10.873703] I [client.c:2090:client_rpc_notify] 37-master-client-0: disconnected
>[2012-06-01 04:21:10.873752] I [client.c:2090:client_rpc_notify] 37-master-client-1: disconnected
>[2012-06-01 04:21:10.873766] E [afr-common.c:3664:afr_notify] 37-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:10.873790] I [client.c:2090:client_rpc_notify] 37-master-client-2: disconnected
>[2012-06-01 04:21:10.873812] I [client.c:2090:client_rpc_notify] 37-master-client-3: disconnected
>[2012-06-01 04:21:10.873823] E [afr-common.c:3664:afr_notify] 37-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:11.167142] I [io-cache.c:1549:check_cache_size_ok] 39-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:11.167241] I [io-cache.c:1549:check_cache_size_ok] 39-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:11.167302] I [quota.c:3085:quota_parse_limits] 39-master-quota: no "limit-set" option provided
>[2012-06-01 04:21:11.174682] I [client.c:2142:notify] 39-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:11.179096] I [client.c:2142:notify] 39-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:11.188381] I [client.c:2142:notify] 39-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:11.192787] I [client.c:2142:notify] 39-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:11.203853] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:11.203996] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:11.204107] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:11.204214] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:13.515183] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:14.566351] I [io-cache.c:1549:check_cache_size_ok] 40-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:14.566438] I [io-cache.c:1549:check_cache_size_ok] 40-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:14.696669] I [client.c:2142:notify] 40-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:14.707709] I [client.c:2142:notify] 40-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:14.721787] I [client.c:2142:notify] 40-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:14.726648] I [client.c:2142:notify] 40-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:14.740256] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:14.740563] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:14.740666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:14.740954] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:14.835874] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:14.836967] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:14.836987] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:14.837101] I [afr-common.c:3627:afr_notify] 39-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:14.837710] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-0: Server lk version = 1
>[2012-06-01 04:21:14.840149] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:14.841603] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:14.841643] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:14.842883] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-1: Server lk version = 1
>[2012-06-01 04:21:14.852486] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:14.855474] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:14.855495] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:14.855608] I [afr-common.c:3627:afr_notify] 39-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:14.857465] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-2: Server lk version = 1
>[2012-06-01 04:21:14.858490] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:14.859345] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:14.859365] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:14.868687] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 39
>[2012-06-01 04:21:14.869814] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-3: Server lk version = 1
>[2012-06-01 04:21:14.870325] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 39-master-replicate-0: added root inode
>[2012-06-01 04:21:14.873223] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 39-master-replicate-1: added root inode
>[2012-06-01 04:21:14.900131] I [client.c:2151:notify] 38-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:14.900183] I [client.c:2151:notify] 38-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:14.900210] I [client.c:2151:notify] 38-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:14.900234] I [client.c:2151:notify] 38-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:14.905044] I [client.c:2090:client_rpc_notify] 38-master-client-0: disconnected
>[2012-06-01 04:21:14.905081] I [client.c:2090:client_rpc_notify] 38-master-client-1: disconnected
>[2012-06-01 04:21:14.905094] E [afr-common.c:3664:afr_notify] 38-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:14.905115] I [client.c:2090:client_rpc_notify] 38-master-client-2: disconnected
>[2012-06-01 04:21:14.905138] I [client.c:2090:client_rpc_notify] 38-master-client-3: disconnected
>[2012-06-01 04:21:14.905148] E [afr-common.c:3664:afr_notify] 38-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:17.140304] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:17.867835] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:17.868411] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:17.869977] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:17.869998] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:17.870115] I [afr-common.c:3627:afr_notify] 40-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:17.871598] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-0: Server lk version = 1
>[2012-06-01 04:21:17.871633] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:17.871647] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:17.872166] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-1: Server lk version = 1
>[2012-06-01 04:21:17.881267] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:17.882077] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:17.882100] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:17.882700] I [afr-common.c:3627:afr_notify] 40-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:17.885268] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-2: Server lk version = 1
>[2012-06-01 04:21:17.893307] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:17.894461] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:17.894492] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:17.923460] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 40
>[2012-06-01 04:21:17.928309] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-3: Server lk version = 1
>[2012-06-01 04:21:17.931119] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 40-master-replicate-0: added root inode
>[2012-06-01 04:21:17.932271] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 40-master-replicate-1: added root inode
>[2012-06-01 04:21:17.951788] W [quota.c:2377:quota_fsync_cbk] 39-master-quota: quota context not set in inode (gfid:8056796e-06f9-4de2-8d4e-9cd1e1e22b21)
>[2012-06-01 04:21:17.958549] I [client.c:2151:notify] 39-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:17.958609] I [client.c:2151:notify] 39-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:17.958657] I [client.c:2151:notify] 39-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:17.958681] I [client.c:2151:notify] 39-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:17.961774] I [client.c:2090:client_rpc_notify] 39-master-client-0: disconnected
>[2012-06-01 04:21:17.961810] I [client.c:2090:client_rpc_notify] 39-master-client-1: disconnected
>[2012-06-01 04:21:17.961823] E [afr-common.c:3664:afr_notify] 39-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:17.961845] I [client.c:2090:client_rpc_notify] 39-master-client-2: disconnected
>[2012-06-01 04:21:17.961867] I [client.c:2090:client_rpc_notify] 39-master-client-3: disconnected
>[2012-06-01 04:21:17.961878] E [afr-common.c:3664:afr_notify] 39-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:18.375171] I [io-cache.c:1549:check_cache_size_ok] 41-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:18.375299] I [io-cache.c:1549:check_cache_size_ok] 41-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:18.375362] I [quota.c:3085:quota_parse_limits] 41-master-quota: no "limit-set" option provided
>[2012-06-01 04:21:18.407008] I [client.c:2142:notify] 41-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:18.409447] I [client.c:2142:notify] 41-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:18.418114] I [client.c:2142:notify] 41-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:18.428746] I [client.c:2142:notify] 41-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:18.454419] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:18.454511] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:18.454585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:18.454678] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:20.666556] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:21.766005] I [io-cache.c:1549:check_cache_size_ok] 42-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:21.766082] I [io-cache.c:1549:check_cache_size_ok] 42-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:21.792719] I [client.c:2142:notify] 42-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:21.804276] I [client.c:2142:notify] 42-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:21.815859] I [client.c:2142:notify] 42-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:21.829650] I [client.c:2142:notify] 42-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:21.845992] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:21.846656] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:21.846978] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:21.847409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:21.905424] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:21.915351] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:21.915381] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:21.916014] I [afr-common.c:3627:afr_notify] 41-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:21.916123] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-0: Server lk version = 1
>[2012-06-01 04:21:21.921296] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:21.924777] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:21.924799] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:21.927125] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-1: Server lk version = 1
>[2012-06-01 04:21:21.954798] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:21.955794] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:21.955815] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:21.955970] I [afr-common.c:3627:afr_notify] 41-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:21.957800] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-2: Server lk version = 1
>[2012-06-01 04:21:21.998397] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:21.999570] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:21.999595] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:22.106387] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 41
>[2012-06-01 04:21:22.107219] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-3: Server lk version = 1
>[2012-06-01 04:21:22.113256] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 41-master-replicate-0: added root inode
>[2012-06-01 04:21:22.114079] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 41-master-replicate-1: added root inode
>[2012-06-01 04:21:22.199203] I [client.c:2151:notify] 40-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:22.199263] I [client.c:2151:notify] 40-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:22.199292] I [client.c:2151:notify] 40-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:22.199318] I [client.c:2151:notify] 40-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:22.205047] I [client.c:2090:client_rpc_notify] 40-master-client-0: disconnected
>[2012-06-01 04:21:22.205089] I [client.c:2090:client_rpc_notify] 40-master-client-1: disconnected
>[2012-06-01 04:21:22.205102] E [afr-common.c:3664:afr_notify] 40-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:22.205126] I [client.c:2090:client_rpc_notify] 40-master-client-2: disconnected
>[2012-06-01 04:21:22.205150] I [client.c:2090:client_rpc_notify] 40-master-client-3: disconnected
>[2012-06-01 04:21:22.205161] E [afr-common.c:3664:afr_notify] 40-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:24.404385] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:24.987761] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:24.988650] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:24.988671] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:24.988785] I [afr-common.c:3627:afr_notify] 42-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:24.991442] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-0: Server lk version = 1
>[2012-06-01 04:21:24.997476] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:25.003386] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:25.003410] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:25.003992] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:25.004675] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-1: Server lk version = 1
>[2012-06-01 04:21:25.004713] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:25.004769] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:25.004888] I [afr-common.c:3627:afr_notify] 42-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:25.010445] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-2: Server lk version = 1
>[2012-06-01 04:21:25.013538] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:25.014198] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:25.014219] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:25.030828] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 42
>[2012-06-01 04:21:25.031347] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-3: Server lk version = 1
>[2012-06-01 04:21:25.036076] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 42-master-replicate-0: added root inode
>[2012-06-01 04:21:25.037504] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 42-master-replicate-1: added root inode
>[2012-06-01 04:21:25.042519] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:665c9158-d394-4999-88fc-f134a1f4422e)
>[2012-06-01 04:21:25.047014] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:d58a6538-1af4-41eb-ae3b-0e6ef60dd97c)
>[2012-06-01 04:21:25.051878] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:9cd8b3d8-7e7a-45c0-8b3d-a2600e946953)
>[2012-06-01 04:21:25.055540] I [client.c:2151:notify] 41-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:25.055589] I [client.c:2151:notify] 41-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:25.055633] I [client.c:2151:notify] 41-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:25.055662] I [client.c:2151:notify] 41-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:25.059690] I [client.c:2090:client_rpc_notify] 41-master-client-0: disconnected
>[2012-06-01 04:21:25.059733] I [client.c:2090:client_rpc_notify] 41-master-client-1: disconnected
>[2012-06-01 04:21:25.059747] E [afr-common.c:3664:afr_notify] 41-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:25.059771] I [client.c:2090:client_rpc_notify] 41-master-client-2: disconnected
>[2012-06-01 04:21:25.059798] I [client.c:2090:client_rpc_notify] 41-master-client-3: disconnected
>[2012-06-01 04:21:25.059809] E [afr-common.c:3664:afr_notify] 41-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:25.610050] I [io-cache.c:1549:check_cache_size_ok] 43-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:25.610122] I [io-cache.c:1549:check_cache_size_ok] 43-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:25.610190] I [quota.c:3085:quota_parse_limits] 43-master-quota: no "limit-set" option provided
>[2012-06-01 04:21:25.636506] I [client.c:2142:notify] 43-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:25.642832] I [client.c:2142:notify] 43-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:25.650635] I [client.c:2142:notify] 43-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:25.655432] I [client.c:2142:notify] 43-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:25.665336] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:25.665418] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:25.665492] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:25.667277] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:28.101498] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:29.016654] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:29.026402] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:29.026424] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:29.028079] I [afr-common.c:3627:afr_notify] 43-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:29.030045] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-0: Server lk version = 1
>[2012-06-01 04:21:29.031087] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:29.033092] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:29.033112] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:29.033953] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:29.034840] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-1: Server lk version = 1
>[2012-06-01 04:21:29.034892] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:29.034911] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:29.035140] I [afr-common.c:3627:afr_notify] 43-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:29.036482] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-2: Server lk version = 1
>[2012-06-01 04:21:29.051775] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:29.052718] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:29.052741] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:29.092151] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 43
>[2012-06-01 04:21:29.093676] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-3: Server lk version = 1
>[2012-06-01 04:21:29.096785] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 43-master-replicate-0: added root inode
>[2012-06-01 04:21:29.098080] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 43-master-replicate-1: added root inode
>[2012-06-01 04:21:29.110498] I [client.c:2151:notify] 42-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:29.110681] I [client.c:2151:notify] 42-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:29.110850] I [client.c:2151:notify] 42-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:29.110977] I [client.c:2151:notify] 42-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:29.112011] I [socket.c:1798:socket_event_handler] 42-transport: disconnecting now
>[2012-06-01 04:21:29.112083] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 42-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:21:29.107297 (xid=0x2530x)
>[2012-06-01 04:21:29.112105] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 42-master-client-1: remote operation failed: Transport endpoint is not connected
>[2012-06-01 04:21:29.112133] I [client.c:2090:client_rpc_notify] 42-master-client-1: disconnected
>[2012-06-01 04:21:29.112162] I [client.c:2090:client_rpc_notify] 42-master-client-2: disconnected
>[2012-06-01 04:21:29.112184] I [client.c:2090:client_rpc_notify] 42-master-client-0: disconnected
>[2012-06-01 04:21:29.112204] E [afr-common.c:3664:afr_notify] 42-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:29.112473] I [client.c:2090:client_rpc_notify] 42-master-client-3: disconnected
>[2012-06-01 04:21:29.112492] E [afr-common.c:3664:afr_notify] 42-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:34.261159] I [io-cache.c:1549:check_cache_size_ok] 44-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:34.261240] I [io-cache.c:1549:check_cache_size_ok] 44-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:34.345146] I [client.c:2142:notify] 44-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:34.350750] I [client.c:2142:notify] 44-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:34.356755] I [client.c:2142:notify] 44-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:34.362546] I [client.c:2142:notify] 44-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:34.392398] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:34.394364] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:34.394452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:34.396954] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:36.738499] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:37.913315] I [io-cache.c:1549:check_cache_size_ok] 45-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:37.913387] I [io-cache.c:1549:check_cache_size_ok] 45-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:37.913443] I [quota.c:3085:quota_parse_limits] 45-master-quota: no "limit-set" option provided
>[2012-06-01 04:21:37.924646] I [client.c:2142:notify] 45-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:37.932405] I [client.c:2142:notify] 45-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:37.937839] I [client.c:2142:notify] 45-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:37.945231] I [client.c:2142:notify] 45-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:37.956134] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:37.956372] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:37.956466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:37.956779] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:38.058756] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:38.059803] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:38.059825] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:38.060342] I [afr-common.c:3627:afr_notify] 44-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:38.061851] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-0: Server lk version = 1
>[2012-06-01 04:21:38.066559] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:38.070469] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:38.070492] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:38.080383] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-1: Server lk version = 1
>[2012-06-01 04:21:38.081807] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:38.085458] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:38.085481] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:38.085525] I [afr-common.c:3627:afr_notify] 44-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:38.089110] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-2: Server lk version = 1
>[2012-06-01 04:21:38.130237] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:38.132662] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:38.132689] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:38.146217] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 44
>[2012-06-01 04:21:38.146386] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-3: Server lk version = 1
>[2012-06-01 04:21:38.151578] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 44-master-replicate-0: added root inode
>[2012-06-01 04:21:38.152447] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 44-master-replicate-1: added root inode
>[2012-06-01 04:21:38.165401] I [client.c:2151:notify] 43-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:38.165457] I [client.c:2151:notify] 43-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:38.165489] I [client.c:2151:notify] 43-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:38.165526] I [client.c:2151:notify] 43-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:38.169125] I [client.c:2090:client_rpc_notify] 43-master-client-0: disconnected
>[2012-06-01 04:21:38.169163] I [client.c:2090:client_rpc_notify] 43-master-client-1: disconnected
>[2012-06-01 04:21:38.169176] E [afr-common.c:3664:afr_notify] 43-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:38.169199] I [client.c:2090:client_rpc_notify] 43-master-client-2: disconnected
>[2012-06-01 04:21:38.169222] I [client.c:2090:client_rpc_notify] 43-master-client-3: disconnected
>[2012-06-01 04:21:38.169234] E [afr-common.c:3664:afr_notify] 43-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:41.121839] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:41.124034] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:41.124054] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:41.124171] I [afr-common.c:3627:afr_notify] 45-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:21:41.126296] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-0: Server lk version = 1
>[2012-06-01 04:21:41.126344] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:41.126584] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:41.126604] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:41.132366] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-1: Server lk version = 1
>[2012-06-01 04:21:41.132399] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:41.134858] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:41.134881] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:41.135002] I [afr-common.c:3627:afr_notify] 45-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:41.135646] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-2: Server lk version = 1
>[2012-06-01 04:21:41.137018] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:41.140117] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:41.140139] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:41.191542] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 45
>[2012-06-01 04:21:41.192949] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-3: Server lk version = 1
>[2012-06-01 04:21:41.200215] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 45-master-replicate-0: added root inode
>[2012-06-01 04:21:41.205793] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 45-master-replicate-1: added root inode
>[2012-06-01 04:21:41.230667] I [client.c:2151:notify] 44-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:41.230720] I [client.c:2151:notify] 44-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:41.230747] I [client.c:2151:notify] 44-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:41.230772] I [client.c:2151:notify] 44-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:41.234100] I [client.c:2090:client_rpc_notify] 44-master-client-0: disconnected
>[2012-06-01 04:21:41.234134] I [client.c:2090:client_rpc_notify] 44-master-client-1: disconnected
>[2012-06-01 04:21:41.234147] E [afr-common.c:3664:afr_notify] 44-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:41.234168] I [client.c:2090:client_rpc_notify] 44-master-client-2: disconnected
>[2012-06-01 04:21:41.234677] I [client.c:2090:client_rpc_notify] 44-master-client-3: disconnected
>[2012-06-01 04:21:41.234694] E [afr-common.c:3664:afr_notify] 44-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:45.567087] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:21:46.658736] I [io-cache.c:1549:check_cache_size_ok] 46-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:21:46.658811] I [io-cache.c:1549:check_cache_size_ok] 46-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:21:46.700798] I [client.c:2142:notify] 46-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:46.713654] I [client.c:2142:notify] 46-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:46.716364] I [client.c:2142:notify] 46-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:21:46.723926] I [client.c:2142:notify] 46-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:21:46.746385] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:21:46.746880] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:21:46.747108] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:21:46.747914] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:21:50.149578] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:50.151123] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:50.157343] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:21:50.157372] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:50.159043] I [afr-common.c:3627:afr_notify] 46-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:21:50.159420] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:50.159483] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:21:50.159499] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:50.163456] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-1: Server lk version = 1
>[2012-06-01 04:21:50.163494] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:21:50.163511] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:50.163681] I [afr-common.c:3627:afr_notify] 46-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:21:50.163713] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-0: Server lk version = 1
>[2012-06-01 04:21:50.165767] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-2: Server lk version = 1
>[2012-06-01 04:21:50.167902] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:21:50.169360] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:21:50.169381] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:21:50.196255] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 46
>[2012-06-01 04:21:50.200728] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-3: Server lk version = 1
>[2012-06-01 04:21:50.215713] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 46-master-replicate-0: added root inode
>[2012-06-01 04:21:50.217188] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 46-master-replicate-1: added root inode
>[2012-06-01 04:21:50.226915] W [quota.c:2377:quota_fsync_cbk] 45-master-quota: quota context not set in inode (gfid:99e7a5c0-6e3f-4b38-a81a-8e11cb242402)
>[2012-06-01 04:21:50.274021] W [quota.c:2377:quota_fsync_cbk] 45-master-quota: quota context not set in inode (gfid:ae1bc032-d549-4d9c-a084-6d8750427624)
>[2012-06-01 04:21:50.277566] I [client.c:2151:notify] 45-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:50.277644] I [client.c:2151:notify] 45-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:50.277678] I [client.c:2151:notify] 45-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:50.277710] I [client.c:2151:notify] 45-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:21:50.282164] I [client.c:2090:client_rpc_notify] 45-master-client-0: disconnected
>[2012-06-01 04:21:50.282200] I [client.c:2090:client_rpc_notify] 45-master-client-1: disconnected
>[2012-06-01 04:21:50.282212] E [afr-common.c:3664:afr_notify] 45-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:50.282233] I [client.c:2090:client_rpc_notify] 45-master-client-2: disconnected
>[2012-06-01 04:21:50.282254] I [client.c:2090:client_rpc_notify] 45-master-client-3: disconnected
>[2012-06-01 04:21:50.282265] E [afr-common.c:3664:afr_notify] 45-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:21:54.243496] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:05.812391] I [io-cache.c:1549:check_cache_size_ok] 47-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:05.812463] I [io-cache.c:1549:check_cache_size_ok] 47-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:05.812518] I [quota.c:3085:quota_parse_limits] 47-master-quota: no "limit-set" option provided
>[2012-06-01 04:22:05.817725] I [client.c:2142:notify] 47-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:05.828587] I [client.c:2142:notify] 47-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:05.831151] I [client.c:2142:notify] 47-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:05.839937] I [client.c:2142:notify] 47-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:05.854559] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:05.854659] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:05.854736] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:05.854856] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:09.190722] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:09.191371] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:09.191546] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:09.191712] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:09.192286] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:09.192307] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:09.192421] I [afr-common.c:3627:afr_notify] 47-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:09.192457] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:09.192470] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:09.192834] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:09.192854] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:09.192991] I [afr-common.c:3627:afr_notify] 47-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:09.193026] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:09.193047] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:09.202624] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 47
>[2012-06-01 04:22:09.203140] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-0: Server lk version = 1
>[2012-06-01 04:22:09.203172] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-1: Server lk version = 1
>[2012-06-01 04:22:09.203193] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-2: Server lk version = 1
>[2012-06-01 04:22:09.203213] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-3: Server lk version = 1
>[2012-06-01 04:22:09.207689] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 47-master-replicate-0: added root inode
>[2012-06-01 04:22:09.221711] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 47-master-replicate-1: added root inode
>[2012-06-01 04:22:09.332959] I [client.c:2151:notify] 46-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:09.333029] I [client.c:2151:notify] 46-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:09.333060] I [client.c:2151:notify] 46-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:09.333086] I [client.c:2151:notify] 46-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:09.338281] I [client.c:2090:client_rpc_notify] 46-master-client-0: disconnected
>[2012-06-01 04:22:09.338319] I [client.c:2090:client_rpc_notify] 46-master-client-1: disconnected
>[2012-06-01 04:22:09.338332] E [afr-common.c:3664:afr_notify] 46-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:09.338354] I [client.c:2090:client_rpc_notify] 46-master-client-2: disconnected
>[2012-06-01 04:22:09.338378] I [client.c:2090:client_rpc_notify] 46-master-client-3: disconnected
>[2012-06-01 04:22:09.338387] E [afr-common.c:3664:afr_notify] 46-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:09.453100] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:10.499759] I [io-cache.c:1549:check_cache_size_ok] 48-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:10.499838] I [io-cache.c:1549:check_cache_size_ok] 48-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:10.507399] I [client.c:2142:notify] 48-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:10.511312] I [client.c:2142:notify] 48-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:10.514026] I [client.c:2142:notify] 48-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:10.516435] I [client.c:2142:notify] 48-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:10.523300] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:10.523415] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:10.523550] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:10.523649] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:13.304083] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:14.205212] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:14.205629] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:14.205655] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:14.205800] I [afr-common.c:3627:afr_notify] 48-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:14.205928] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-0: Server lk version = 1
>[2012-06-01 04:22:14.219049] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:14.219953] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:14.219980] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:14.222207] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-1: Server lk version = 1
>[2012-06-01 04:22:14.226570] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:14.227814] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:14.227847] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:14.228019] I [afr-common.c:3627:afr_notify] 48-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:14.229728] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-2: Server lk version = 1
>[2012-06-01 04:22:14.234203] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:14.234454] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:14.234478] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:14.241344] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 48
>[2012-06-01 04:22:14.241793] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-3: Server lk version = 1
>[2012-06-01 04:22:14.243819] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 48-master-replicate-0: added root inode
>[2012-06-01 04:22:14.244830] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 48-master-replicate-1: added root inode
>[2012-06-01 04:22:14.340120] W [quota.c:2377:quota_fsync_cbk] 47-master-quota: quota context not set in inode (gfid:a643fd8b-c2fd-4d3a-8dc6-efb7683c3cbb)
>[2012-06-01 04:22:14.405808] W [quota.c:2377:quota_fsync_cbk] 47-master-quota: quota context not set in inode (gfid:9cd8b3d8-7e7a-45c0-8b3d-a2600e946953)
>[2012-06-01 04:22:14.408276] I [client.c:2151:notify] 47-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:14.408341] I [client.c:2151:notify] 47-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:14.408377] I [client.c:2151:notify] 47-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:14.408408] I [client.c:2151:notify] 47-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:14.412441] I [client.c:2090:client_rpc_notify] 47-master-client-0: disconnected
>[2012-06-01 04:22:14.412500] I [client.c:2090:client_rpc_notify] 47-master-client-1: disconnected
>[2012-06-01 04:22:14.412530] E [afr-common.c:3664:afr_notify] 47-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:14.412564] I [client.c:2090:client_rpc_notify] 47-master-client-2: disconnected
>[2012-06-01 04:22:14.412594] I [client.c:2090:client_rpc_notify] 47-master-client-3: disconnected
>[2012-06-01 04:22:14.412609] E [afr-common.c:3664:afr_notify] 47-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:14.855986] I [io-cache.c:1549:check_cache_size_ok] 49-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:14.856128] I [io-cache.c:1549:check_cache_size_ok] 49-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:14.856193] I [quota.c:3085:quota_parse_limits] 49-master-quota: no "limit-set" option provided
>[2012-06-01 04:22:14.866508] I [client.c:2142:notify] 49-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:14.871607] I [client.c:2142:notify] 49-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:14.876360] I [client.c:2142:notify] 49-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:14.882274] I [client.c:2142:notify] 49-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:14.896991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:14.897229] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:14.897349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:14.897466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:17.948743] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:18.247197] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:18.248985] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:18.249011] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:18.249149] I [afr-common.c:3627:afr_notify] 49-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:18.249864] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-0: Server lk version = 1
>[2012-06-01 04:22:18.249911] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:18.250921] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:18.250947] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:18.253553] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-1: Server lk version = 1
>[2012-06-01 04:22:18.261997] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:18.262227] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:18.263105] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:18.263130] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:18.263265] I [afr-common.c:3627:afr_notify] 49-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:18.263587] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:18.263610] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:18.272771] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 49
>[2012-06-01 04:22:18.273038] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-2: Server lk version = 1
>[2012-06-01 04:22:18.273072] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-3: Server lk version = 1
>[2012-06-01 04:22:18.276074] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 49-master-replicate-0: added root inode
>[2012-06-01 04:22:18.277458] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 49-master-replicate-1: added root inode
>[2012-06-01 04:22:18.335357] I [client.c:2151:notify] 48-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:18.335433] I [client.c:2151:notify] 48-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:18.335470] I [client.c:2151:notify] 48-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:18.335500] I [client.c:2151:notify] 48-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:18.336280] I [client.c:2090:client_rpc_notify] 48-master-client-0: disconnected
>[2012-06-01 04:22:18.336371] I [client.c:2090:client_rpc_notify] 48-master-client-1: disconnected
>[2012-06-01 04:22:18.336400] E [afr-common.c:3664:afr_notify] 48-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:18.336430] I [client.c:2090:client_rpc_notify] 48-master-client-2: disconnected
>[2012-06-01 04:22:18.336459] I [client.c:2090:client_rpc_notify] 48-master-client-3: disconnected
>[2012-06-01 04:22:18.336473] E [afr-common.c:3664:afr_notify] 48-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:19.081019] I [io-cache.c:1549:check_cache_size_ok] 50-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:19.081130] I [io-cache.c:1549:check_cache_size_ok] 50-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:19.171562] I [client.c:2142:notify] 50-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:19.182372] I [client.c:2142:notify] 50-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:19.187046] I [client.c:2142:notify] 50-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:19.205318] I [client.c:2142:notify] 50-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:19.215356] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:19.215504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:19.215666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:19.215785] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:21.401169] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:22.270596] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:22.270823] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:22.272315] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:22.272341] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:22.272485] I [afr-common.c:3627:afr_notify] 50-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:22:22.273080] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:22.273100] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:22.273930] I [afr-common.c:3627:afr_notify] 50-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:22.274323] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:22.275594] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-1: Server lk version = 1
>[2012-06-01 04:22:22.275716] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-2: Server lk version = 1
>[2012-06-01 04:22:22.276638] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:22.276659] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:22.279138] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-0: Server lk version = 1
>[2012-06-01 04:22:22.291120] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:22.292076] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:22.292097] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:22.315236] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 50
>[2012-06-01 04:22:22.316248] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-3: Server lk version = 1
>[2012-06-01 04:22:22.325531] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 50-master-replicate-0: added root inode
>[2012-06-01 04:22:22.326907] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 50-master-replicate-1: added root inode
>[2012-06-01 04:22:22.498424] I [client.c:2151:notify] 49-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:22.498508] I [client.c:2151:notify] 49-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:22.498554] I [client.c:2151:notify] 49-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:22.498595] I [client.c:2151:notify] 49-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:22.503137] I [client.c:2090:client_rpc_notify] 49-master-client-0: disconnected
>[2012-06-01 04:22:22.503196] I [client.c:2090:client_rpc_notify] 49-master-client-1: disconnected
>[2012-06-01 04:22:22.503217] E [afr-common.c:3664:afr_notify] 49-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:22.503256] I [client.c:2090:client_rpc_notify] 49-master-client-2: disconnected
>[2012-06-01 04:22:22.503294] I [client.c:2090:client_rpc_notify] 49-master-client-3: disconnected
>[2012-06-01 04:22:22.503313] E [afr-common.c:3664:afr_notify] 49-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:22.771824] I [io-cache.c:1549:check_cache_size_ok] 51-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:22.771953] I [io-cache.c:1549:check_cache_size_ok] 51-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:22.772053] I [quota.c:3085:quota_parse_limits] 51-master-quota: no "limit-set" option provided
>[2012-06-01 04:22:22.784070] I [client.c:2142:notify] 51-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:22.794895] I [client.c:2142:notify] 51-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:22.809660] I [client.c:2142:notify] 51-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:22.821020] I [client.c:2142:notify] 51-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:22.894579] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:22.897524] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:22.898234] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:22.898666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:26.299825] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:26.300265] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:26.300765] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:26.300787] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:26.301254] I [afr-common.c:3627:afr_notify] 51-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:26.301294] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:26.301309] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:26.306631] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-0: Server lk version = 1
>[2012-06-01 04:22:26.307362] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-1: Server lk version = 1
>[2012-06-01 04:22:26.322418] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:26.323784] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:26.324410] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:26.325686] I [afr-common.c:3627:afr_notify] 51-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:26.327858] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-2: Server lk version = 1
>[2012-06-01 04:22:26.340816] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:26.343182] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:26.343228] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:26.353854] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 51
>[2012-06-01 04:22:26.355004] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-3: Server lk version = 1
>[2012-06-01 04:22:26.362228] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 51-master-replicate-0: added root inode
>[2012-06-01 04:22:26.365429] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 51-master-replicate-1: added root inode
>[2012-06-01 04:22:26.390160] I [client.c:2151:notify] 50-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:26.390226] I [client.c:2151:notify] 50-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:26.390260] I [client.c:2151:notify] 50-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:26.390288] I [client.c:2151:notify] 50-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:26.397075] I [client.c:2090:client_rpc_notify] 50-master-client-0: disconnected
>[2012-06-01 04:22:26.397115] I [client.c:2090:client_rpc_notify] 50-master-client-1: disconnected
>[2012-06-01 04:22:26.397130] E [afr-common.c:3664:afr_notify] 50-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:26.397164] I [client.c:2090:client_rpc_notify] 50-master-client-2: disconnected
>[2012-06-01 04:22:26.397190] I [client.c:2090:client_rpc_notify] 50-master-client-3: disconnected
>[2012-06-01 04:22:26.397202] E [afr-common.c:3664:afr_notify] 50-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:30.958870] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:32.145307] I [io-cache.c:1549:check_cache_size_ok] 52-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:32.145378] I [io-cache.c:1549:check_cache_size_ok] 52-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:32.208777] I [client.c:2142:notify] 52-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:32.211943] I [client.c:2142:notify] 52-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:32.217839] I [client.c:2142:notify] 52-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:32.223433] I [client.c:2142:notify] 52-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:32.231450] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:32.231652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:32.231814] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:32.231991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:35.352501] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:35.353443] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:35.353465] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:35.353583] I [afr-common.c:3627:afr_notify] 52-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:35.355963] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-0: Server lk version = 1
>[2012-06-01 04:22:35.356507] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:35.357004] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:35.357023] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:35.357339] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-1: Server lk version = 1
>[2012-06-01 04:22:35.382089] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:35.383645] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:35.383666] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:35.384606] I [afr-common.c:3627:afr_notify] 52-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:35.385504] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-2: Server lk version = 1
>[2012-06-01 04:22:35.393689] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:35.395756] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:35.395779] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:35.428479] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 52
>[2012-06-01 04:22:35.428781] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-3: Server lk version = 1
>[2012-06-01 04:22:35.436528] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 52-master-replicate-0: added root inode
>[2012-06-01 04:22:35.437928] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 52-master-replicate-1: added root inode
>[2012-06-01 04:22:35.462261] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:a43e29f2-2d14-4191-8ee7-0507690c8b76)
>[2012-06-01 04:22:35.468760] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b)
>[2012-06-01 04:22:35.479247] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:05d26344-33d0-4f42-80d6-a3dd2695fb2b)
>[2012-06-01 04:22:35.487828] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:561277dc-03e2-4127-baee-b556fc9799f3)
>[2012-06-01 04:22:35.492668] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:81f303f5-e0c4-419a-8cb5-a808cbd1015e)
>[2012-06-01 04:22:35.495528] I [client.c:2151:notify] 51-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:35.495585] I [client.c:2151:notify] 51-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:35.495618] I [client.c:2151:notify] 51-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:35.495643] I [client.c:2151:notify] 51-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:35.499592] I [client.c:2090:client_rpc_notify] 51-master-client-0: disconnected
>[2012-06-01 04:22:35.499630] I [client.c:2090:client_rpc_notify] 51-master-client-1: disconnected
>[2012-06-01 04:22:35.499644] E [afr-common.c:3664:afr_notify] 51-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:35.499668] I [client.c:2090:client_rpc_notify] 51-master-client-2: disconnected
>[2012-06-01 04:22:35.499690] I [client.c:2090:client_rpc_notify] 51-master-client-3: disconnected
>[2012-06-01 04:22:35.499719] E [afr-common.c:3664:afr_notify] 51-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:35.625775] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:37.070914] I [io-cache.c:1549:check_cache_size_ok] 53-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:37.071040] I [io-cache.c:1549:check_cache_size_ok] 53-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:37.071142] I [quota.c:3085:quota_parse_limits] 53-master-quota: no "limit-set" option provided
>[2012-06-01 04:22:37.227877] I [client.c:2142:notify] 53-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:37.237168] I [client.c:2142:notify] 53-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:37.248704] I [client.c:2142:notify] 53-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:37.276149] I [client.c:2142:notify] 53-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:37.323031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:37.323337] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:37.323552] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:37.323712] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:40.404417] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:40.408601] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:40.408663] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:40.408848] I [afr-common.c:3627:afr_notify] 53-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:40.409353] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-0: Server lk version = 1
>[2012-06-01 04:22:40.427523] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:40.435910] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:40.435946] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:40.437732] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-1: Server lk version = 1
>[2012-06-01 04:22:40.458480] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:40.460446] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:40.460480] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:40.460718] I [afr-common.c:3627:afr_notify] 53-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:40.462492] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-2: Server lk version = 1
>[2012-06-01 04:22:40.488228] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:40.491173] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:40.491208] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:40.524895] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 53
>[2012-06-01 04:22:40.531081] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-3: Server lk version = 1
>[2012-06-01 04:22:40.538997] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 53-master-replicate-0: added root inode
>[2012-06-01 04:22:40.541472] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 53-master-replicate-1: added root inode
>[2012-06-01 04:22:40.602319] I [client.c:2151:notify] 52-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:40.602394] I [client.c:2151:notify] 52-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:40.602437] I [client.c:2151:notify] 52-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:40.602478] I [client.c:2151:notify] 52-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:40.609290] I [client.c:2090:client_rpc_notify] 52-master-client-0: disconnected
>[2012-06-01 04:22:40.609332] I [client.c:2090:client_rpc_notify] 52-master-client-1: disconnected
>[2012-06-01 04:22:40.609348] E [afr-common.c:3664:afr_notify] 52-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:40.610214] I [client.c:2090:client_rpc_notify] 52-master-client-2: disconnected
>[2012-06-01 04:22:40.610247] I [client.c:2090:client_rpc_notify] 52-master-client-3: disconnected
>[2012-06-01 04:22:40.610259] E [afr-common.c:3664:afr_notify] 52-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:40.981009] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:47.321224] I [io-cache.c:1549:check_cache_size_ok] 54-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:47.321355] I [io-cache.c:1549:check_cache_size_ok] 54-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:47.416007] I [client.c:2142:notify] 54-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:47.454686] I [client.c:2142:notify] 54-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:47.460190] I [client.c:2142:notify] 54-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:47.462666] I [client.c:2142:notify] 54-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:47.477502] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:47.477922] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:47.484022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:47.484643] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:50.520752] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:50.539139] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:50.540479] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:50.540509] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:50.540583] I [afr-common.c:3627:afr_notify] 54-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:50.542445] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:50.542468] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:50.544903] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-1: Server lk version = 1
>[2012-06-01 04:22:50.544937] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-0: Server lk version = 1
>[2012-06-01 04:22:51.544810] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:51.546671] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:51.546694] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:51.546815] I [afr-common.c:3627:afr_notify] 54-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:22:51.546849] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:51.548579] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-2: Server lk version = 1
>[2012-06-01 04:22:51.548619] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:51.548629] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:51.578266] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 54
>[2012-06-01 04:22:51.578558] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-3: Server lk version = 1
>[2012-06-01 04:22:51.580395] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 54-master-replicate-0: added root inode
>[2012-06-01 04:22:51.581802] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 54-master-replicate-1: added root inode
>[2012-06-01 04:22:51.592892] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:23b92615-8b00-4f7d-919b-dd30d836b438)
>[2012-06-01 04:22:51.646954] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:09b17a28-6b3d-4fe1-bb09-e7caefdef1ce)
>[2012-06-01 04:22:51.654440] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:d55db600-df01-40a6-852b-f66cd5332de7)
>[2012-06-01 04:22:51.669529] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:161164b1-58e3-4eef-9864-0ffcb97521c0)
>[2012-06-01 04:22:51.696840] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:8bab20b7-696b-4ba6-91c6-68c7ddd96714)
>[2012-06-01 04:22:51.700746] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:c906f6b4-7e84-4ac5-a6f0-536b45dd9b4f)
>[2012-06-01 04:22:51.710118] I [client.c:2151:notify] 53-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:51.710249] I [client.c:2151:notify] 53-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:51.710293] I [client.c:2151:notify] 53-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:51.710324] I [client.c:2151:notify] 53-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:51.727570] I [client.c:2090:client_rpc_notify] 53-master-client-0: disconnected
>[2012-06-01 04:22:51.727638] I [client.c:2090:client_rpc_notify] 53-master-client-1: disconnected
>[2012-06-01 04:22:51.727656] E [afr-common.c:3664:afr_notify] 53-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:51.727872] I [client.c:2090:client_rpc_notify] 53-master-client-2: disconnected
>[2012-06-01 04:22:51.727914] I [client.c:2090:client_rpc_notify] 53-master-client-3: disconnected
>[2012-06-01 04:22:51.727927] E [afr-common.c:3664:afr_notify] 53-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:52.860325] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:22:54.702128] I [io-cache.c:1549:check_cache_size_ok] 55-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:22:54.702231] I [io-cache.c:1549:check_cache_size_ok] 55-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:22:54.702319] I [quota.c:3085:quota_parse_limits] 55-master-quota: no "limit-set" option provided
>[2012-06-01 04:22:55.008186] I [client.c:2142:notify] 55-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:55.036223] I [client.c:2142:notify] 55-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:55.052030] I [client.c:2142:notify] 55-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:22:55.061427] I [client.c:2142:notify] 55-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:22:55.116773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:22:55.117016] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:22:55.117142] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:22:55.117271] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:22:58.549959] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:58.550663] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:22:58.550680] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:58.550801] I [afr-common.c:3627:afr_notify] 55-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:22:58.551134] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-0: Server lk version = 1
>[2012-06-01 04:22:58.551172] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:58.557787] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:22:58.557798] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:58.557944] I [afr-common.c:3627:afr_notify] 55-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:22:58.558660] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-3: Server lk version = 1
>[2012-06-01 04:22:58.593402] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:58.593905] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:22:58.593923] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:58.594955] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-1: Server lk version = 1
>[2012-06-01 04:22:58.620628] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:22:58.633137] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:22:58.633166] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:22:58.741674] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 55
>[2012-06-01 04:22:58.742530] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-2: Server lk version = 1
>[2012-06-01 04:22:58.750076] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 55-master-replicate-0: added root inode
>[2012-06-01 04:22:58.751027] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 55-master-replicate-1: added root inode
>[2012-06-01 04:22:59.039433] I [client.c:2151:notify] 54-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:59.039545] I [client.c:2151:notify] 54-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:59.039612] I [client.c:2151:notify] 54-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:59.039684] I [client.c:2151:notify] 54-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:22:59.135979] I [client.c:2090:client_rpc_notify] 54-master-client-0: disconnected
>[2012-06-01 04:22:59.136680] I [client.c:2090:client_rpc_notify] 54-master-client-1: disconnected
>[2012-06-01 04:22:59.139826] E [afr-common.c:3664:afr_notify] 54-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:22:59.139892] I [client.c:2090:client_rpc_notify] 54-master-client-2: disconnected
>[2012-06-01 04:22:59.142019] I [client.c:2090:client_rpc_notify] 54-master-client-3: disconnected
>[2012-06-01 04:22:59.142049] E [afr-common.c:3664:afr_notify] 54-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:00.275492] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:01.687540] I [io-cache.c:1549:check_cache_size_ok] 56-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:01.687654] I [io-cache.c:1549:check_cache_size_ok] 56-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:01.701416] I [client.c:2142:notify] 56-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:01.708544] I [client.c:2142:notify] 56-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:01.714124] I [client.c:2142:notify] 56-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:01.718149] I [client.c:2142:notify] 56-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:01.732764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:01.732893] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:01.733166] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:01.733253] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:05.637029] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:05.641574] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:05.641960] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:05.641981] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:05.642047] I [afr-common.c:3627:afr_notify] 56-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:05.642080] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:05.642093] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:05.642320] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-0: Server lk version = 1
>[2012-06-01 04:23:05.642349] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-1: Server lk version = 1
>[2012-06-01 04:23:05.642367] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:05.642808] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:05.642835] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:05.642879] I [afr-common.c:3627:afr_notify] 56-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:05.644162] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-2: Server lk version = 1
>[2012-06-01 04:23:05.670760] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:05.672346] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:05.672368] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:05.679819] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 56
>[2012-06-01 04:23:05.683623] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-3: Server lk version = 1
>[2012-06-01 04:23:05.685037] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 56-master-replicate-0: added root inode
>[2012-06-01 04:23:05.691379] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 56-master-replicate-1: added root inode
>[2012-06-01 04:23:05.692820] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:05.695112] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:a97a35c3-a7aa-4b3c-b400-50164d53446f)
>[2012-06-01 04:23:05.793497] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b)
>[2012-06-01 04:23:05.796189] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:b2374a2f-cfed-4df8-a691-ace626e2e6b4)
>[2012-06-01 04:23:05.811259] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:d0e3460f-c940-4dc1-b991-839d48e3965e)
>[2012-06-01 04:23:05.825710] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:debaad44-cdff-483e-b38c-6d04a346efb9)
>[2012-06-01 04:23:05.836399] I [client.c:2151:notify] 55-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:05.836473] I [client.c:2151:notify] 55-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:05.836514] I [client.c:2151:notify] 55-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:05.836552] I [client.c:2151:notify] 55-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:05.856491] I [client.c:2090:client_rpc_notify] 55-master-client-0: disconnected
>[2012-06-01 04:23:05.856536] I [client.c:2090:client_rpc_notify] 55-master-client-1: disconnected
>[2012-06-01 04:23:05.856551] E [afr-common.c:3664:afr_notify] 55-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:05.856580] I [client.c:2090:client_rpc_notify] 55-master-client-2: disconnected
>[2012-06-01 04:23:05.856606] I [client.c:2090:client_rpc_notify] 55-master-client-3: disconnected
>[2012-06-01 04:23:05.856644] E [afr-common.c:3664:afr_notify] 55-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:07.039103] I [io-cache.c:1549:check_cache_size_ok] 57-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:07.039196] I [io-cache.c:1549:check_cache_size_ok] 57-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:07.039267] I [quota.c:3085:quota_parse_limits] 57-master-quota: no "limit-set" option provided
>[2012-06-01 04:23:07.113931] I [client.c:2142:notify] 57-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:07.134929] I [client.c:2142:notify] 57-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:07.142599] I [client.c:2142:notify] 57-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:07.153168] I [client.c:2142:notify] 57-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:07.167989] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:07.169500] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:07.169600] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:07.169676] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:10.115128] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:10.679050] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:10.679142] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:10.684360] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:10.684383] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:10.684522] I [afr-common.c:3627:afr_notify] 57-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:10.684559] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:10.684572] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:10.684692] I [afr-common.c:3627:afr_notify] 57-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:10.685009] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-0: Server lk version = 1
>[2012-06-01 04:23:10.685045] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-2: Server lk version = 1
>[2012-06-01 04:23:10.685376] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:10.687190] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:10.687212] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:10.689660] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-3: Server lk version = 1
>[2012-06-01 04:23:10.726906] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:10.733457] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:10.736803] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:10.758662] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 57
>[2012-06-01 04:23:10.759406] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-1: Server lk version = 1
>[2012-06-01 04:23:10.759655] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 57-master-replicate-0: added root inode
>[2012-06-01 04:23:10.764060] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 57-master-replicate-1: added root inode
>[2012-06-01 04:23:10.959817] I [client.c:2151:notify] 56-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:10.959920] I [client.c:2151:notify] 56-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:10.959975] I [client.c:2151:notify] 56-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:10.960020] I [client.c:2151:notify] 56-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:10.974253] I [client.c:2090:client_rpc_notify] 56-master-client-0: disconnected
>[2012-06-01 04:23:10.974292] I [client.c:2090:client_rpc_notify] 56-master-client-1: disconnected
>[2012-06-01 04:23:10.974304] E [afr-common.c:3664:afr_notify] 56-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:10.974330] I [client.c:2090:client_rpc_notify] 56-master-client-2: disconnected
>[2012-06-01 04:23:10.974354] I [client.c:2090:client_rpc_notify] 56-master-client-3: disconnected
>[2012-06-01 04:23:10.974367] E [afr-common.c:3664:afr_notify] 56-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:11.198235] I [io-cache.c:1549:check_cache_size_ok] 58-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:11.198336] I [io-cache.c:1549:check_cache_size_ok] 58-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:11.449432] I [client.c:2142:notify] 58-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:11.485453] I [client.c:2142:notify] 58-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:11.500915] I [client.c:2142:notify] 58-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:11.513585] I [client.c:2142:notify] 58-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:11.533803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:11.534197] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:11.534276] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:11.534356] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:14.750121] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:14.752597] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:14.752693] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:14.753702] I [afr-common.c:3627:afr_notify] 58-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:14.753758] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:14.765873] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-0: Server lk version = 1
>[2012-06-01 04:23:14.765929] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:14.765952] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:14.770042] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-1: Server lk version = 1
>[2012-06-01 04:23:14.781016] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:14.789010] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:14.791539] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:14.791792] I [afr-common.c:3627:afr_notify] 58-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:14.794133] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-2: Server lk version = 1
>[2012-06-01 04:23:14.794184] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:14.803123] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:14.803204] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:14.815822] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 58
>[2012-06-01 04:23:14.820484] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-3: Server lk version = 1
>[2012-06-01 04:23:14.824728] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 58-master-replicate-0: added root inode
>[2012-06-01 04:23:14.827545] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 58-master-replicate-1: added root inode
>[2012-06-01 04:23:14.905884] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:9e934735-4d2d-48e5-b1e5-2af76431ab24)
>[2012-06-01 04:23:14.933230] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:948ce967-4e02-4ce0-aef8-a10ffd7ab165)
>[2012-06-01 04:23:14.939682] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:ffeac997-bf44-4a78-aed1-142f2cd9c1f4)
>[2012-06-01 04:23:14.961065] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:df424240-d25c-44cb-96fe-f87e1c7fb2ab)
>[2012-06-01 04:23:15.009204] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:caa8bd7b-2bd7-4dd2-b5c3-4517a80b5be8)
>[2012-06-01 04:23:15.031164] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:94bc5eda-6b69-40c0-94fc-6f95c47a7693)
>[2012-06-01 04:23:15.042789] I [client.c:2151:notify] 57-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:15.042842] I [client.c:2151:notify] 57-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:15.042905] I [client.c:2151:notify] 57-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:15.042947] I [client.c:2151:notify] 57-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:23:15.066580] I [client.c:2090:client_rpc_notify] 57-master-client-0: disconnected
>[2012-06-01 04:23:15.066624] I [client.c:2090:client_rpc_notify] 57-master-client-1: disconnected
>[2012-06-01 04:23:15.066637] E [afr-common.c:3664:afr_notify] 57-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:15.066664] I [client.c:2090:client_rpc_notify] 57-master-client-2: disconnected
>[2012-06-01 04:23:15.066692] I [client.c:2090:client_rpc_notify] 57-master-client-3: disconnected
>[2012-06-01 04:23:15.066703] E [afr-common.c:3664:afr_notify] 57-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:23:15.070649] W [fuse-resolve.c:359:fuse_resolve_fd] 0-fuse-resolve: migration of fd (0x1b1bce94) did not complete, failing fop with EBADF
>[2012-06-01 04:23:15.074827] W [fuse-resolve.c:359:fuse_resolve_fd] 0-fuse-resolve: migration of fd (0x1b1bce94) did not complete, failing fop with EBADF
>[2012-06-01 04:23:18.953460] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:20.177752] I [io-cache.c:1549:check_cache_size_ok] 59-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:20.177846] I [io-cache.c:1549:check_cache_size_ok] 59-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:20.191406] I [quota.c:3085:quota_parse_limits] 59-master-quota: no "limit-set" option provided
>[2012-06-01 04:23:20.325798] I [client.c:2142:notify] 59-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:20.348541] I [client.c:2142:notify] 59-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:20.361327] I [client.c:2142:notify] 59-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:20.378620] I [client.c:2142:notify] 59-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:20.398119] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:20.398243] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:20.398349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:20.398452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:22.800525] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:23.787160] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:23.787369] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:23.787389] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:23.787434] I [afr-common.c:3627:afr_notify] 59-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:23.787507] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-0: Server lk version = 1
>[2012-06-01 04:23:23.790686] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:23.791231] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:23.791255] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:23.791563] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-1: Server lk version = 1
>[2012-06-01 04:23:23.796046] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:23.796218] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:23.796236] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:23.796279] I [afr-common.c:3627:afr_notify] 59-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:23.796589] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-2: Server lk version = 1
>[2012-06-01 04:23:23.802313] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:23.802484] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:23.802503] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:23.807233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 59
>[2012-06-01 04:23:23.807303] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-3: Server lk version = 1
>[2012-06-01 04:23:23.858155] I [io-cache.c:1549:check_cache_size_ok] 60-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:23.858232] I [io-cache.c:1549:check_cache_size_ok] 60-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:23.927219] I [client.c:2142:notify] 60-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:23.939294] I [client.c:2142:notify] 60-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:23.961461] I [client.c:2142:notify] 60-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:23.964976] I [client.c:2142:notify] 60-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:23.993186] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:23.993236] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:23.993291] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:23.993326] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:27.066517] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:27.818374] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:27.819242] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:27.819277] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:27.819356] I [afr-common.c:3627:afr_notify] 60-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:27.819494] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-0: Server lk version = 1
>[2012-06-01 04:23:27.838318] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:27.838592] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:27.838651] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:27.839059] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-1: Server lk version = 1
>[2012-06-01 04:23:27.889598] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:27.891685] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:27.891716] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:27.891787] I [afr-common.c:3627:afr_notify] 60-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:27.892342] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-2: Server lk version = 1
>[2012-06-01 04:23:27.896011] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:27.896441] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:27.896474] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:27.917302] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 60
>[2012-06-01 04:23:27.917472] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-3: Server lk version = 1
>[2012-06-01 04:23:28.336396] I [io-cache.c:1549:check_cache_size_ok] 61-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:28.336469] I [io-cache.c:1549:check_cache_size_ok] 61-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:28.336541] I [quota.c:3085:quota_parse_limits] 61-master-quota: no "limit-set" option provided
>[2012-06-01 04:23:28.440919] I [client.c:2142:notify] 61-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:28.449815] I [client.c:2142:notify] 61-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:28.485291] I [client.c:2142:notify] 61-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:28.500472] I [client.c:2142:notify] 61-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:28.590237] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:28.590357] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:28.598415] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:28.598942] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:31.935489] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:31.936002] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:31.936094] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:31.936700] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:31.936735] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:31.936845] I [afr-common.c:3627:afr_notify] 61-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:31.936900] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:31.936923] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:31.936996] I [afr-common.c:3627:afr_notify] 61-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:31.937046] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:31.937067] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:31.937230] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-0: Server lk version = 1
>[2012-06-01 04:23:31.937272] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:31.937379] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-1: Server lk version = 1
>[2012-06-01 04:23:31.938687] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-2: Server lk version = 1
>[2012-06-01 04:23:31.939049] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:31.939084] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:31.955826] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 61
>[2012-06-01 04:23:31.955969] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-3: Server lk version = 1
>[2012-06-01 04:23:33.076297] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:34.369814] I [io-cache.c:1549:check_cache_size_ok] 62-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:34.370435] I [io-cache.c:1549:check_cache_size_ok] 62-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:34.396917] I [client.c:2142:notify] 62-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:34.431748] I [client.c:2142:notify] 62-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:34.436939] I [client.c:2142:notify] 62-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:34.450319] I [client.c:2142:notify] 62-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:34.458156] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:34.458291] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:34.458411] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:34.458527] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:37.927490] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:37.933271] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:37.939296] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:37.944405] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:37.953615] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:37.953650] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:37.953729] I [afr-common.c:3627:afr_notify] 62-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:23:37.953782] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:37.953804] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:37.954074] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:37.954105] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:37.954180] I [afr-common.c:3627:afr_notify] 62-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:37.954230] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-1: Server lk version = 1
>[2012-06-01 04:23:37.954263] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-0: Server lk version = 1
>[2012-06-01 04:23:37.954943] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-2: Server lk version = 1
>[2012-06-01 04:23:37.954997] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:37.955019] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:37.961474] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 62
>[2012-06-01 04:23:37.961605] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-3: Server lk version = 1
>[2012-06-01 04:23:38.650978] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:39.916817] I [io-cache.c:1549:check_cache_size_ok] 63-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:39.917680] I [io-cache.c:1549:check_cache_size_ok] 63-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:39.917788] I [quota.c:3085:quota_parse_limits] 63-master-quota: no "limit-set" option provided
>[2012-06-01 04:23:39.925727] I [client.c:2142:notify] 63-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:39.931935] I [client.c:2142:notify] 63-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:39.945128] I [client.c:2142:notify] 63-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:39.950025] I [client.c:2142:notify] 63-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:39.972067] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:39.972219] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:39.972491] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:39.973452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:42.926305] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:43.980124] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:43.980415] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:43.980445] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:43.980533] I [afr-common.c:3627:afr_notify] 63-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:43.980667] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-0: Server lk version = 1
>[2012-06-01 04:23:44.005839] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:44.006176] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:44.006215] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:44.006509] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-1: Server lk version = 1
>[2012-06-01 04:23:44.046779] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:44.047106] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:44.047135] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:44.047208] I [afr-common.c:3627:afr_notify] 63-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:44.047793] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-2: Server lk version = 1
>[2012-06-01 04:23:44.063726] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:44.064032] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:44.064061] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:44.430936] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 63
>[2012-06-01 04:23:44.439518] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-3: Server lk version = 1
>[2012-06-01 04:23:44.451653] I [io-cache.c:1549:check_cache_size_ok] 64-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:44.451772] I [io-cache.c:1549:check_cache_size_ok] 64-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:44.512719] I [client.c:2142:notify] 64-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:44.530315] I [client.c:2142:notify] 64-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:44.546265] I [client.c:2142:notify] 64-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:44.549638] I [client.c:2142:notify] 64-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:44.558393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:44.567135] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:44.567226] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:44.567498] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:46.759311] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:47.952788] I [io-cache.c:1549:check_cache_size_ok] 65-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:47.952882] I [io-cache.c:1549:check_cache_size_ok] 65-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:47.952946] I [quota.c:3085:quota_parse_limits] 65-master-quota: no "limit-set" option provided
>[2012-06-01 04:23:47.978703] I [client.c:2142:notify] 65-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:47.983081] I [client.c:2142:notify] 65-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:48.011713] I [client.c:2142:notify] 65-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:48.015011] I [client.c:2142:notify] 65-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:48.036031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:48.036115] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:48.036153] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:48.036234] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:48.114377] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:48.114573] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:48.114592] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:48.114645] I [afr-common.c:3627:afr_notify] 64-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:48.114736] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-0: Server lk version = 1
>[2012-06-01 04:23:48.144354] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:48.144562] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:48.144581] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:48.144797] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-1: Server lk version = 1
>[2012-06-01 04:23:48.166150] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:48.166348] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:48.166371] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:48.166425] I [afr-common.c:3627:afr_notify] 64-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:48.166523] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-2: Server lk version = 1
>[2012-06-01 04:23:48.169741] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:48.170021] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:48.170040] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:48.184018] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 64
>[2012-06-01 04:23:48.184136] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-3: Server lk version = 1
>[2012-06-01 04:23:50.761456] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:23:51.193536] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:51.193772] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:51.193793] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:51.193840] I [afr-common.c:3627:afr_notify] 65-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:51.194541] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-0: Server lk version = 1
>[2012-06-01 04:23:51.199293] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:51.199558] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:51.199578] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:51.199946] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-1: Server lk version = 1
>[2012-06-01 04:23:51.206187] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:51.206377] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:51.206395] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:51.206440] I [afr-common.c:3627:afr_notify] 65-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:51.206520] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-2: Server lk version = 1
>[2012-06-01 04:23:51.219464] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:51.219911] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:51.219933] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:51.224259] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 65
>[2012-06-01 04:23:51.224332] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-3: Server lk version = 1
>[2012-06-01 04:23:51.823547] I [io-cache.c:1549:check_cache_size_ok] 66-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:23:51.823644] I [io-cache.c:1549:check_cache_size_ok] 66-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:23:51.854023] I [client.c:2142:notify] 66-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:51.863120] I [client.c:2142:notify] 66-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:51.865895] I [client.c:2142:notify] 66-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:23:51.872757] I [client.c:2142:notify] 66-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:23:51.879482] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:23:51.879573] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:23:51.879668] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:23:51.879752] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:23:55.226793] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:55.227425] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:23:55.227447] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:55.227506] I [afr-common.c:3627:afr_notify] 66-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:23:55.227767] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-0: Server lk version = 1
>[2012-06-01 04:23:55.234619] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:55.234831] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:23:55.234852] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:55.234988] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-1: Server lk version = 1
>[2012-06-01 04:23:55.245157] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:55.245337] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:23:55.245354] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:55.245406] I [afr-common.c:3627:afr_notify] 66-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:23:55.245490] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-2: Server lk version = 1
>[2012-06-01 04:23:55.264308] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:23:55.264481] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:23:55.264500] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:23:55.276804] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 66
>[2012-06-01 04:23:55.276903] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-3: Server lk version = 1
>[2012-06-01 04:23:59.542531] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:00.871284] I [io-cache.c:1549:check_cache_size_ok] 67-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:00.871362] I [io-cache.c:1549:check_cache_size_ok] 67-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:00.871430] I [quota.c:3085:quota_parse_limits] 67-master-quota: no "limit-set" option provided
>[2012-06-01 04:24:00.940341] I [client.c:2142:notify] 67-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:00.969935] I [client.c:2142:notify] 67-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:00.978532] I [client.c:2142:notify] 67-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:00.983891] I [client.c:2142:notify] 67-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:00.995193] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:00.995389] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:00.995707] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:00.996226] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:03.548313] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:04.292406] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:04.297857] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:04.302458] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:04.305576] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:04.307603] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:04.307647] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:04.307707] I [afr-common.c:3627:afr_notify] 67-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:24:04.307742] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:04.307765] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:04.307814] I [afr-common.c:3627:afr_notify] 67-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:04.307848] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:04.307863] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:04.308930] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-0: Server lk version = 1
>[2012-06-01 04:24:04.308962] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-1: Server lk version = 1
>[2012-06-01 04:24:04.308987] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:04.309001] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:04.329332] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 67
>[2012-06-01 04:24:04.337001] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-2: Server lk version = 1
>[2012-06-01 04:24:04.337639] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-3: Server lk version = 1
>[2012-06-01 04:24:09.880551] I [io-cache.c:1549:check_cache_size_ok] 68-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:09.881101] I [io-cache.c:1549:check_cache_size_ok] 68-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:09.890389] I [client.c:2142:notify] 68-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:09.911164] I [client.c:2142:notify] 68-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:09.927614] I [client.c:2142:notify] 68-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:09.934342] I [client.c:2142:notify] 68-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:09.939514] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:09.939844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:09.939962] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:09.940072] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:13.320227] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:13.320529] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:13.320559] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:13.320680] I [afr-common.c:3627:afr_notify] 68-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:13.320827] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-0: Server lk version = 1
>[2012-06-01 04:24:13.336061] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:13.336339] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:13.336368] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:13.336801] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-1: Server lk version = 1
>[2012-06-01 04:24:13.352824] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:13.353340] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:13.353371] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:13.353446] I [afr-common.c:3627:afr_notify] 68-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:13.353910] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-2: Server lk version = 1
>[2012-06-01 04:24:13.373035] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:13.373492] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:13.373546] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:13.399266] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 68
>[2012-06-01 04:24:13.399443] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-3: Server lk version = 1
>[2012-06-01 04:24:13.785493] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:20.264522] I [io-cache.c:1549:check_cache_size_ok] 69-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:20.264662] I [io-cache.c:1549:check_cache_size_ok] 69-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:20.264764] I [quota.c:3085:quota_parse_limits] 69-master-quota: no "limit-set" option provided
>[2012-06-01 04:24:20.286678] I [client.c:2142:notify] 69-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:20.309071] I [client.c:2142:notify] 69-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:20.330002] I [client.c:2142:notify] 69-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:20.347913] I [client.c:2142:notify] 69-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:20.374683] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:20.374828] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:20.374979] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:20.377026] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:23.349216] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:23.383624] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:23.383939] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:23.383969] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:23.384044] I [afr-common.c:3627:afr_notify] 69-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:23.384179] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-0: Server lk version = 1
>[2012-06-01 04:24:23.402561] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:23.402972] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:23.403004] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:23.403419] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-1: Server lk version = 1
>[2012-06-01 04:24:23.410779] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:23.411081] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:23.411112] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:23.411188] I [afr-common.c:3627:afr_notify] 69-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:23.411329] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-2: Server lk version = 1
>[2012-06-01 04:24:23.416842] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:23.417404] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:23.417437] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:23.423810] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 69
>[2012-06-01 04:24:23.423911] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-3: Server lk version = 1
>[2012-06-01 04:24:24.432406] I [io-cache.c:1549:check_cache_size_ok] 70-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:24.432520] I [io-cache.c:1549:check_cache_size_ok] 70-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:24.444110] I [client.c:2142:notify] 70-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:24.453436] I [client.c:2142:notify] 70-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:24.466152] I [client.c:2142:notify] 70-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:24.475366] I [client.c:2142:notify] 70-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:24.489290] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:24.489416] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:24.489565] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:24.489661] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:28.425592] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:28.425901] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:28.425927] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:28.425991] I [afr-common.c:3627:afr_notify] 70-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:28.426111] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-0: Server lk version = 1
>[2012-06-01 04:24:28.435409] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:28.435645] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:28.435672] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:28.435732] I [afr-common.c:3627:afr_notify] 70-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:24:28.436159] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-3: Server lk version = 1
>[2012-06-01 04:24:28.444167] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:28.444430] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:28.444456] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:28.444646] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-1: Server lk version = 1
>[2012-06-01 04:24:28.452166] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:28.452388] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:28.452413] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:28.488674] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 70
>[2012-06-01 04:24:28.488762] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-2: Server lk version = 1
>[2012-06-01 04:24:41.690071] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:42.744452] I [io-cache.c:1549:check_cache_size_ok] 71-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:42.744563] I [io-cache.c:1549:check_cache_size_ok] 71-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:42.748731] I [quota.c:3085:quota_parse_limits] 71-master-quota: no "limit-set" option provided
>[2012-06-01 04:24:42.765500] I [client.c:2142:notify] 71-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:42.774609] I [client.c:2142:notify] 71-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:42.781572] I [client.c:2142:notify] 71-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:42.786857] I [client.c:2142:notify] 71-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:42.792765] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:42.792904] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:42.793025] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:42.793142] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:44.883893] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:45.923204] I [io-cache.c:1549:check_cache_size_ok] 72-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:45.923323] I [io-cache.c:1549:check_cache_size_ok] 72-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:45.933817] I [client.c:2142:notify] 72-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:45.944548] I [client.c:2142:notify] 72-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:45.956970] I [client.c:2142:notify] 72-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:45.968641] I [client.c:2142:notify] 72-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:45.976115] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:45.976242] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:45.976360] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:45.976475] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:46.460585] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:46.467991] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:46.468156] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:46.468195] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:46.468310] I [afr-common.c:3627:afr_notify] 71-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:46.468771] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:46.468815] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:46.469231] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-0: Server lk version = 1
>[2012-06-01 04:24:46.469793] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-1: Server lk version = 1
>[2012-06-01 04:24:46.472828] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:46.473469] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:46.473509] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:46.473665] I [afr-common.c:3627:afr_notify] 71-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:46.473790] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-2: Server lk version = 1
>[2012-06-01 04:24:46.478513] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:46.478917] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:46.478959] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:46.484938] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 71
>[2012-06-01 04:24:46.485042] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-3: Server lk version = 1
>[2012-06-01 04:24:48.133466] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:49.292867] I [io-cache.c:1549:check_cache_size_ok] 73-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:49.292990] I [io-cache.c:1549:check_cache_size_ok] 73-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:49.293087] I [quota.c:3085:quota_parse_limits] 73-master-quota: no "limit-set" option provided
>[2012-06-01 04:24:49.397896] I [client.c:2142:notify] 73-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:49.407980] I [client.c:2142:notify] 73-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:49.415854] I [client.c:2142:notify] 73-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:49.430161] I [client.c:2142:notify] 73-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:49.445195] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:49.445339] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:49.445481] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:49.445608] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:24:49.483382] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:49.483736] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:49.483769] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:49.483849] I [afr-common.c:3627:afr_notify] 72-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:49.483991] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-0: Server lk version = 1
>[2012-06-01 04:24:49.493492] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:49.493823] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:49.493854] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:49.494325] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-1: Server lk version = 1
>[2012-06-01 04:24:49.507639] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:49.508003] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:49.508035] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:49.508118] I [afr-common.c:3627:afr_notify] 72-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:49.508312] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-2: Server lk version = 1
>[2012-06-01 04:24:49.519163] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:49.519515] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:49.519556] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:49.737163] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 72
>[2012-06-01 04:24:49.739092] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-3: Server lk version = 1
>[2012-06-01 04:24:51.798057] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:24:52.533018] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:52.533364] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:24:52.533393] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:52.533487] I [afr-common.c:3627:afr_notify] 73-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:24:52.533701] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-0: Server lk version = 1
>[2012-06-01 04:24:52.541060] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:52.541474] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:24:52.541505] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:52.543116] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-1: Server lk version = 1
>[2012-06-01 04:24:52.553315] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:52.553603] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:24:52.553665] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:52.553740] I [afr-common.c:3627:afr_notify] 73-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:24:52.553874] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-2: Server lk version = 1
>[2012-06-01 04:24:52.561282] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:24:52.561944] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:24:52.561976] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:24:52.566802] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 73
>[2012-06-01 04:24:52.566899] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-3: Server lk version = 1
>[2012-06-01 04:24:57.851374] I [io-cache.c:1549:check_cache_size_ok] 74-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:24:57.851473] I [io-cache.c:1549:check_cache_size_ok] 74-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:24:57.864647] I [client.c:2142:notify] 74-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:57.879965] I [client.c:2142:notify] 74-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:57.887730] I [client.c:2142:notify] 74-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:24:57.902645] I [client.c:2142:notify] 74-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:24:57.912818] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:24:57.913022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:24:57.913210] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:24:57.913600] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:00.171243] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:01.287018] I [io-cache.c:1549:check_cache_size_ok] 75-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:01.287095] I [io-cache.c:1549:check_cache_size_ok] 75-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:01.287157] I [quota.c:3085:quota_parse_limits] 75-master-quota: no "limit-set" option provided
>[2012-06-01 04:25:01.364532] I [client.c:2142:notify] 75-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:01.375417] I [client.c:2142:notify] 75-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:01.386708] I [client.c:2142:notify] 75-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:01.401540] I [client.c:2142:notify] 75-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:01.415129] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:01.415196] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:01.415237] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:01.415324] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:01.574028] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:01.574252] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:01.574271] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:01.574328] I [afr-common.c:3627:afr_notify] 74-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:01.574443] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-0: Server lk version = 1
>[2012-06-01 04:25:01.579116] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:01.579327] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:01.579351] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:01.579536] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-1: Server lk version = 1
>[2012-06-01 04:25:01.584065] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:01.584577] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:01.584597] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:01.584656] I [afr-common.c:3627:afr_notify] 74-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:01.584684] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-2: Server lk version = 1
>[2012-06-01 04:25:01.588763] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:01.589031] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:01.589051] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:01.598620] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 74
>[2012-06-01 04:25:01.598745] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-3: Server lk version = 1
>[2012-06-01 04:25:03.621923] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:04.602551] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:04.602799] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:04.602824] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:04.602892] I [afr-common.c:3627:afr_notify] 75-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:04.602985] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-0: Server lk version = 1
>[2012-06-01 04:25:04.607242] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:04.607574] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:04.607593] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:04.608019] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-1: Server lk version = 1
>[2012-06-01 04:25:04.615045] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:04.615236] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:04.615257] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:04.615299] I [afr-common.c:3627:afr_notify] 75-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:04.615380] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-2: Server lk version = 1
>[2012-06-01 04:25:04.621555] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:04.629151] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:04.629171] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:04.634621] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 75
>[2012-06-01 04:25:04.634690] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-3: Server lk version = 1
>[2012-06-01 04:25:04.667501] I [io-cache.c:1549:check_cache_size_ok] 76-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:04.667627] I [io-cache.c:1549:check_cache_size_ok] 76-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:04.696535] I [client.c:2142:notify] 76-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:04.702678] I [client.c:2142:notify] 76-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:04.708247] I [client.c:2142:notify] 76-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:04.713981] I [client.c:2142:notify] 76-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:04.720405] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:04.720494] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:04.720571] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:04.720670] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:07.157475] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:08.269436] I [io-cache.c:1549:check_cache_size_ok] 77-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:08.269537] I [io-cache.c:1549:check_cache_size_ok] 77-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:08.269615] I [quota.c:3085:quota_parse_limits] 77-master-quota: no "limit-set" option provided
>[2012-06-01 04:25:08.303204] I [client.c:2142:notify] 77-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:08.322124] I [client.c:2142:notify] 77-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:08.333204] I [client.c:2142:notify] 77-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:08.348814] I [client.c:2142:notify] 77-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:08.365536] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:08.365905] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:08.366387] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:08.366999] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:08.626832] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:08.627092] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:08.627116] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:08.627174] I [afr-common.c:3627:afr_notify] 76-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:08.627276] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-0: Server lk version = 1
>[2012-06-01 04:25:08.634345] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:08.634564] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:08.634587] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:08.635271] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-1: Server lk version = 1
>[2012-06-01 04:25:08.640701] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:08.640955] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:08.640978] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:08.641033] I [afr-common.c:3627:afr_notify] 76-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:08.641148] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-2: Server lk version = 1
>[2012-06-01 04:25:08.648018] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:08.649001] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:08.649028] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:08.657901] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 76
>[2012-06-01 04:25:08.657981] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-3: Server lk version = 1
>[2012-06-01 04:25:10.782022] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:11.654492] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:11.655059] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:11.655094] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:11.655175] I [afr-common.c:3627:afr_notify] 77-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:11.657282] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-0: Server lk version = 1
>[2012-06-01 04:25:11.665166] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:11.665906] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:11.665946] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:11.666246] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-1: Server lk version = 1
>[2012-06-01 04:25:11.691511] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:11.691827] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:11.691857] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:11.692004] I [afr-common.c:3627:afr_notify] 77-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:11.692149] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-2: Server lk version = 1
>[2012-06-01 04:25:11.700978] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:11.701430] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:11.701461] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:11.720711] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 77
>[2012-06-01 04:25:11.720792] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-3: Server lk version = 1
>[2012-06-01 04:25:21.929733] I [io-cache.c:1549:check_cache_size_ok] 78-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:21.929828] I [io-cache.c:1549:check_cache_size_ok] 78-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:21.937189] I [client.c:2142:notify] 78-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:21.941683] I [client.c:2142:notify] 78-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:21.955942] I [client.c:2142:notify] 78-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:21.959760] I [client.c:2142:notify] 78-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:21.964114] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:21.964222] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:21.964317] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:21.964408] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:25.718365] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:25.718645] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:25.718673] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:25.718740] I [afr-common.c:3627:afr_notify] 78-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:25.718859] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-0: Server lk version = 1
>[2012-06-01 04:25:25.725806] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:25.726401] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:25.726430] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:25.726796] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-1: Server lk version = 1
>[2012-06-01 04:25:25.734557] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:25.734815] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:25.734841] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:25.734937] I [afr-common.c:3627:afr_notify] 78-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:25.735059] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-2: Server lk version = 1
>[2012-06-01 04:25:25.740723] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:25.741678] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:25.741706] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:25.764943] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 78
>[2012-06-01 04:25:25.765033] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-3: Server lk version = 1
>[2012-06-01 04:25:34.246855] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:35.313453] I [io-cache.c:1549:check_cache_size_ok] 79-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:35.313581] I [io-cache.c:1549:check_cache_size_ok] 79-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:35.313680] I [quota.c:3085:quota_parse_limits] 79-master-quota: no "limit-set" option provided
>[2012-06-01 04:25:35.321618] I [client.c:2142:notify] 79-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:35.326539] I [client.c:2142:notify] 79-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:35.331120] I [client.c:2142:notify] 79-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:35.335679] I [client.c:2142:notify] 79-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:35.341440] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:35.341551] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:35.341672] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:35.341733] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:37.433487] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:38.468750] I [io-cache.c:1549:check_cache_size_ok] 80-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:38.468852] I [io-cache.c:1549:check_cache_size_ok] 80-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:38.493901] I [client.c:2142:notify] 80-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:38.519342] I [client.c:2142:notify] 80-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:38.533058] I [client.c:2142:notify] 80-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:38.557551] I [client.c:2142:notify] 80-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:38.567232] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:38.567390] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:38.567737] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:38.567931] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:38.756409] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:38.756818] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:38.756849] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:38.757171] I [afr-common.c:3627:afr_notify] 79-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:38.757223] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-0: Server lk version = 1
>[2012-06-01 04:25:38.761815] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:38.762143] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:38.762173] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:38.762612] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-1: Server lk version = 1
>[2012-06-01 04:25:38.767910] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:38.768232] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:38.768261] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:38.768369] I [afr-common.c:3627:afr_notify] 79-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:38.768416] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-2: Server lk version = 1
>[2012-06-01 04:25:38.773137] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:38.773417] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:38.773446] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:38.804441] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 79
>[2012-06-01 04:25:38.804574] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-3: Server lk version = 1
>[2012-06-01 04:25:40.814788] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:41.800000] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:41.800313] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:41.800344] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:41.800421] I [afr-common.c:3627:afr_notify] 80-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:41.800561] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-0: Server lk version = 1
>[2012-06-01 04:25:41.805604] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:41.805928] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:41.805960] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:41.806423] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-1: Server lk version = 1
>[2012-06-01 04:25:41.860726] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:41.861078] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:41.861110] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:41.861188] I [afr-common.c:3627:afr_notify] 80-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:41.861328] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-2: Server lk version = 1
>[2012-06-01 04:25:41.939481] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:41.939816] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:41.939848] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:41.945393] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 80
>[2012-06-01 04:25:41.946307] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-3: Server lk version = 1
>[2012-06-01 04:25:42.081328] I [io-cache.c:1549:check_cache_size_ok] 81-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:42.081448] I [io-cache.c:1549:check_cache_size_ok] 81-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:42.081544] I [quota.c:3085:quota_parse_limits] 81-master-quota: no "limit-set" option provided
>[2012-06-01 04:25:42.190104] I [client.c:2142:notify] 81-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:42.206973] I [client.c:2142:notify] 81-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:42.212141] I [client.c:2142:notify] 81-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:42.227800] I [client.c:2142:notify] 81-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:42.237536] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:42.238122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:42.238261] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:42.238576] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:44.493635] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:45.956320] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:45.956728] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:45.956760] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:45.956861] I [afr-common.c:3627:afr_notify] 81-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:45.956996] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-0: Server lk version = 1
>[2012-06-01 04:25:45.964984] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:45.965447] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:45.965479] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:45.965897] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-1: Server lk version = 1
>[2012-06-01 04:25:45.972997] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:45.990063] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:45.990102] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:45.990208] I [afr-common.c:3627:afr_notify] 81-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:45.990309] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-2: Server lk version = 1
>[2012-06-01 04:25:45.990410] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:45.990757] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:45.990792] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:45.998478] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 81
>[2012-06-01 04:25:45.998577] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-3: Server lk version = 1
>[2012-06-01 04:25:50.571620] I [io-cache.c:1549:check_cache_size_ok] 82-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:50.571731] I [io-cache.c:1549:check_cache_size_ok] 82-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:50.614660] I [client.c:2142:notify] 82-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:50.629212] I [client.c:2142:notify] 82-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:50.638335] I [client.c:2142:notify] 82-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:50.652386] I [client.c:2142:notify] 82-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:50.671479] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:50.671718] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:50.671844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:50.671997] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:52.971334] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:53.984486] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:53.994526] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:53.994553] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:53.994613] I [afr-common.c:3627:afr_notify] 82-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:53.994717] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-0: Server lk version = 1
>[2012-06-01 04:25:53.994769] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:53.994992] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:53.995022] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:53.995409] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-1: Server lk version = 1
>[2012-06-01 04:25:54.000929] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:54.001150] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:54.001172] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:54.001226] I [afr-common.c:3627:afr_notify] 82-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:54.001326] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-2: Server lk version = 1
>[2012-06-01 04:25:54.014542] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:54.014759] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:54.014782] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:54.070842] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 82
>[2012-06-01 04:25:54.070916] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-3: Server lk version = 1
>[2012-06-01 04:25:54.140730] I [io-cache.c:1549:check_cache_size_ok] 83-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:54.140812] I [io-cache.c:1549:check_cache_size_ok] 83-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:54.140884] I [quota.c:3085:quota_parse_limits] 83-master-quota: no "limit-set" option provided
>[2012-06-01 04:25:54.172536] I [client.c:2142:notify] 83-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:54.182149] I [client.c:2142:notify] 83-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:54.188958] I [client.c:2142:notify] 83-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:54.195787] I [client.c:2142:notify] 83-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:54.203379] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:54.203825] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:54.203916] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:54.204002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:25:57.402341] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:25:58.042439] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:58.042857] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:25:58.042935] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:58.043038] I [afr-common.c:3627:afr_notify] 83-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:25:58.043221] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-0: Server lk version = 1
>[2012-06-01 04:25:58.058355] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:58.058647] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:25:58.058677] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:58.059150] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-1: Server lk version = 1
>[2012-06-01 04:25:58.083781] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:58.084257] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:25:58.084305] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:58.084425] I [afr-common.c:3627:afr_notify] 83-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:25:58.084587] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-2: Server lk version = 1
>[2012-06-01 04:25:58.112010] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:25:58.112438] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:25:58.112500] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:25:58.136785] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 83
>[2012-06-01 04:25:58.136913] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-3: Server lk version = 1
>[2012-06-01 04:25:58.553958] I [io-cache.c:1549:check_cache_size_ok] 84-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:25:58.554069] I [io-cache.c:1549:check_cache_size_ok] 84-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:25:58.784168] I [client.c:2142:notify] 84-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:58.803671] I [client.c:2142:notify] 84-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:58.831921] I [client.c:2142:notify] 84-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:25:58.841943] I [client.c:2142:notify] 84-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:25:58.858797] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:25:58.859312] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:25:58.859445] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:25:58.859591] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:02.132125] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:02.142556] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:02.147790] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:02.152918] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:02.171527] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:02.171577] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:02.171726] I [afr-common.c:3627:afr_notify] 84-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:02.171807] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:02.171838] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:02.171952] I [afr-common.c:3627:afr_notify] 84-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:02.172007] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:02.172029] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:02.195879] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-0: Server lk version = 1
>[2012-06-01 04:26:02.196019] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-2: Server lk version = 1
>[2012-06-01 04:26:02.196052] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-1: Server lk version = 1
>[2012-06-01 04:26:02.196092] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:02.196114] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:02.224845] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 84
>[2012-06-01 04:26:02.230254] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-3: Server lk version = 1
>[2012-06-01 04:26:03.397177] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:04.915560] I [io-cache.c:1549:check_cache_size_ok] 85-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:04.915706] I [io-cache.c:1549:check_cache_size_ok] 85-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:04.915826] I [quota.c:3085:quota_parse_limits] 85-master-quota: no "limit-set" option provided
>[2012-06-01 04:26:04.988267] I [client.c:2142:notify] 85-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:05.019972] I [client.c:2142:notify] 85-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:05.053913] I [client.c:2142:notify] 85-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:05.106986] I [client.c:2142:notify] 85-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:05.126674] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:05.126773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:05.139510] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:05.139682] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:08.165445] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:08.173672] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:08.179292] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:08.184521] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:08.201965] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:08.202007] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:08.202118] I [afr-common.c:3627:afr_notify] 85-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:08.202177] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:08.202199] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:08.226573] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:08.226657] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:08.226741] I [afr-common.c:3627:afr_notify] 85-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:26:08.226795] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:08.226817] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:08.237035] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 85
>[2012-06-01 04:26:08.242580] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-1: Server lk version = 1
>[2012-06-01 04:26:08.242630] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-3: Server lk version = 1
>[2012-06-01 04:26:08.242663] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-2: Server lk version = 1
>[2012-06-01 04:26:08.242694] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-0: Server lk version = 1
>[2012-06-01 04:26:10.247056] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:12.637846] I [io-cache.c:1549:check_cache_size_ok] 86-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:12.638073] I [io-cache.c:1549:check_cache_size_ok] 86-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:12.671248] I [client.c:2142:notify] 86-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:12.686194] I [client.c:2142:notify] 86-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:12.696335] I [client.c:2142:notify] 86-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:12.717060] I [client.c:2142:notify] 86-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:12.740225] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:12.740419] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:12.740647] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:12.740810] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:16.092034] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:16.254415] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:16.254751] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:16.254781] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:16.254857] I [afr-common.c:3627:afr_notify] 86-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:16.255035] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-0: Server lk version = 1
>[2012-06-01 04:26:16.265385] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:16.265728] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:16.265758] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:16.266628] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-1: Server lk version = 1
>[2012-06-01 04:26:16.278916] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:16.279286] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:16.279315] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:16.279416] I [afr-common.c:3627:afr_notify] 86-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:16.279557] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-2: Server lk version = 1
>[2012-06-01 04:26:16.285264] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:16.285747] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:16.285779] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:16.291612] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 86
>[2012-06-01 04:26:16.291726] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-3: Server lk version = 1
>[2012-06-01 04:26:17.370763] I [io-cache.c:1549:check_cache_size_ok] 87-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:17.372805] I [io-cache.c:1549:check_cache_size_ok] 87-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:17.372947] I [quota.c:3085:quota_parse_limits] 87-master-quota: no "limit-set" option provided
>[2012-06-01 04:26:17.400938] I [client.c:2142:notify] 87-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:17.417149] I [client.c:2142:notify] 87-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:17.432096] I [client.c:2142:notify] 87-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:17.436617] I [client.c:2142:notify] 87-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:17.464311] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:17.464474] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:17.464598] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:17.467559] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:20.425585] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:21.296042] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:21.296411] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:21.296456] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:21.296536] I [afr-common.c:3627:afr_notify] 87-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:21.296753] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-0: Server lk version = 1
>[2012-06-01 04:26:21.309093] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:21.309371] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:21.309400] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:21.309842] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-1: Server lk version = 1
>[2012-06-01 04:26:21.331229] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:21.331876] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:21.331908] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:21.331987] I [afr-common.c:3627:afr_notify] 87-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:21.332125] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-2: Server lk version = 1
>[2012-06-01 04:26:21.338741] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:21.339062] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:21.339094] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:21.701067] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 87
>[2012-06-01 04:26:21.701166] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-3: Server lk version = 1
>[2012-06-01 04:26:21.703605] I [io-cache.c:1549:check_cache_size_ok] 88-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:21.703721] I [io-cache.c:1549:check_cache_size_ok] 88-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:21.733231] I [client.c:2142:notify] 88-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:21.753931] I [client.c:2142:notify] 88-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:21.769114] I [client.c:2142:notify] 88-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:21.793138] I [client.c:2142:notify] 88-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:21.838410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:21.838585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:21.838717] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:21.838844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:25.361085] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:25.361388] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:25.361422] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:25.361538] I [afr-common.c:3627:afr_notify] 88-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:25.361698] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-0: Server lk version = 1
>[2012-06-01 04:26:25.375068] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:25.375344] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:25.375368] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:25.375598] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-1: Server lk version = 1
>[2012-06-01 04:26:25.383654] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:25.383881] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:25.383902] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:25.383969] I [afr-common.c:3627:afr_notify] 88-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:25.384001] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-2: Server lk version = 1
>[2012-06-01 04:26:25.403093] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:25.403300] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:25.403321] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:25.437227] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 88
>[2012-06-01 04:26:25.458031] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-3: Server lk version = 1
>[2012-06-01 04:26:29.549645] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:30.891176] I [io-cache.c:1549:check_cache_size_ok] 89-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:30.891289] I [io-cache.c:1549:check_cache_size_ok] 89-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:30.891378] I [quota.c:3085:quota_parse_limits] 89-master-quota: no "limit-set" option provided
>[2012-06-01 04:26:30.969162] I [client.c:2142:notify] 89-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:30.991104] I [client.c:2142:notify] 89-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:31.011658] I [client.c:2142:notify] 89-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:31.027545] I [client.c:2142:notify] 89-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:31.051119] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:31.051425] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:31.051667] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:31.051906] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:34.413141] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:34.413904] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:34.413951] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:34.414068] I [afr-common.c:3627:afr_notify] 89-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:34.414173] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-0: Server lk version = 1
>[2012-06-01 04:26:34.440177] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:34.440549] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:34.440579] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:34.441193] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-1: Server lk version = 1
>[2012-06-01 04:26:34.457680] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:34.458364] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:34.458397] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:34.458479] I [afr-common.c:3627:afr_notify] 89-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:34.458810] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-2: Server lk version = 1
>[2012-06-01 04:26:34.506053] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:34.506345] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:34.506375] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:34.511783] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 89
>[2012-06-01 04:26:34.520421] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-3: Server lk version = 1
>[2012-06-01 04:26:34.641960] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:40.925725] I [io-cache.c:1549:check_cache_size_ok] 90-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:40.925849] I [io-cache.c:1549:check_cache_size_ok] 90-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:40.932990] I [client.c:2142:notify] 90-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:40.959942] I [client.c:2142:notify] 90-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:40.964056] I [client.c:2142:notify] 90-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:40.968679] I [client.c:2142:notify] 90-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:40.972157] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:40.972256] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:40.972333] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:40.972409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:44.380546] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:44.519012] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:44.519259] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:44.519278] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:44.519329] I [afr-common.c:3627:afr_notify] 90-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:44.519398] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-0: Server lk version = 1
>[2012-06-01 04:26:44.532470] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:44.532655] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:44.532674] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:44.532948] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-1: Server lk version = 1
>[2012-06-01 04:26:44.562305] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:44.562509] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:44.562529] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:44.562602] I [afr-common.c:3627:afr_notify] 90-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:44.562648] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-2: Server lk version = 1
>[2012-06-01 04:26:44.572870] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:44.573054] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:44.573072] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:44.578166] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 90
>[2012-06-01 04:26:44.578245] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-3: Server lk version = 1
>[2012-06-01 04:26:45.631394] I [io-cache.c:1549:check_cache_size_ok] 91-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:45.631471] I [io-cache.c:1549:check_cache_size_ok] 91-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:45.631549] I [quota.c:3085:quota_parse_limits] 91-master-quota: no "limit-set" option provided
>[2012-06-01 04:26:45.661095] I [client.c:2142:notify] 91-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:45.673166] I [client.c:2142:notify] 91-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:45.680452] I [client.c:2142:notify] 91-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:45.691558] I [client.c:2142:notify] 91-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:45.703035] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:45.703103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:45.703161] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:45.703216] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:48.702252] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:49.585781] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:49.586036] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:49.586057] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:49.586104] I [afr-common.c:3627:afr_notify] 91-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:49.586190] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-0: Server lk version = 1
>[2012-06-01 04:26:49.594565] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:49.594883] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:49.594907] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:49.594950] I [afr-common.c:3627:afr_notify] 91-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:26:49.595026] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-3: Server lk version = 1
>[2012-06-01 04:26:49.597705] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:49.597880] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:49.597898] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:49.598108] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-1: Server lk version = 1
>[2012-06-01 04:26:49.625959] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:49.626160] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:49.626183] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:49.662829] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 91
>[2012-06-01 04:26:49.662930] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-2: Server lk version = 1
>[2012-06-01 04:26:49.809794] I [io-cache.c:1549:check_cache_size_ok] 92-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:49.809896] I [io-cache.c:1549:check_cache_size_ok] 92-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:50.057759] I [client.c:2142:notify] 92-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:50.076242] I [client.c:2142:notify] 92-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:50.116977] I [client.c:2142:notify] 92-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:50.148694] I [client.c:2142:notify] 92-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:50.182765] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:50.182924] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:50.183030] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:50.185716] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:26:53.642929] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:53.643267] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:26:53.643298] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:53.643406] I [afr-common.c:3627:afr_notify] 92-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:26:53.643826] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-0: Server lk version = 1
>[2012-06-01 04:26:53.651557] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:53.651905] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:26:53.651936] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:53.652140] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-1: Server lk version = 1
>[2012-06-01 04:26:53.665424] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:53.665821] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:26:53.665866] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:53.666022] I [afr-common.c:3627:afr_notify] 92-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:26:53.666090] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-2: Server lk version = 1
>[2012-06-01 04:26:53.673156] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:26:53.673421] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:26:53.673451] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:26:53.683125] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 92
>[2012-06-01 04:26:53.683211] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-3: Server lk version = 1
>[2012-06-01 04:26:57.864600] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:26:59.274372] I [io-cache.c:1549:check_cache_size_ok] 93-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:26:59.274502] I [io-cache.c:1549:check_cache_size_ok] 93-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:26:59.274603] I [quota.c:3085:quota_parse_limits] 93-master-quota: no "limit-set" option provided
>[2012-06-01 04:26:59.285782] I [client.c:2142:notify] 93-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:59.297088] I [client.c:2142:notify] 93-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:59.311843] I [client.c:2142:notify] 93-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:26:59.326475] I [client.c:2142:notify] 93-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:26:59.339735] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:26:59.340123] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:26:59.340251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:26:59.340393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:01.862625] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:02.679733] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:02.680000] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:02.680046] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:02.680131] I [afr-common.c:3627:afr_notify] 93-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:02.680169] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-0: Server lk version = 1
>[2012-06-01 04:27:02.691850] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:02.692070] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:02.692094] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:02.692366] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-1: Server lk version = 1
>[2012-06-01 04:27:02.719209] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:02.719530] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:02.719556] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:02.719642] I [afr-common.c:3627:afr_notify] 93-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:02.719879] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-2: Server lk version = 1
>[2012-06-01 04:27:02.736002] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:02.738175] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:02.738206] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:02.745542] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 93
>[2012-06-01 04:27:02.745640] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-3: Server lk version = 1
>[2012-06-01 04:27:02.940824] I [io-cache.c:1549:check_cache_size_ok] 94-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:02.940955] I [io-cache.c:1549:check_cache_size_ok] 94-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:03.009183] I [client.c:2142:notify] 94-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:03.019119] I [client.c:2142:notify] 94-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:03.033799] I [client.c:2142:notify] 94-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:03.046180] I [client.c:2142:notify] 94-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:03.056609] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:03.056773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:03.056883] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:03.056957] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:05.589159] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:06.780598] I [io-cache.c:1549:check_cache_size_ok] 95-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:06.780707] I [io-cache.c:1549:check_cache_size_ok] 95-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:06.780783] I [quota.c:3085:quota_parse_limits] 95-master-quota: no "limit-set" option provided
>[2012-06-01 04:27:06.816102] I [client.c:2142:notify] 95-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:06.826084] I [client.c:2142:notify] 95-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:06.840301] I [client.c:2142:notify] 95-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:06.849057] I [client.c:2142:notify] 95-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:06.860936] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:06.862956] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:06.862983] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:06.863237] I [afr-common.c:3627:afr_notify] 94-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:06.863281] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:06.863646] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:06.872982] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:06.873910] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-0: Server lk version = 1
>[2012-06-01 04:27:06.873956] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:06.873974] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:06.877089] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:06.877132] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:06.877226] I [afr-common.c:3627:afr_notify] 94-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:06.877270] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:06.877289] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:06.892184] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 94
>[2012-06-01 04:27:06.892257] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:06.892408] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-1: Server lk version = 1
>[2012-06-01 04:27:06.892458] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-2: Server lk version = 1
>[2012-06-01 04:27:06.892484] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-3: Server lk version = 1
>[2012-06-01 04:27:06.892529] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:06.892633] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:06.892741] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:09.309512] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:10.413225] I [io-cache.c:1549:check_cache_size_ok] 96-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:10.413332] I [io-cache.c:1549:check_cache_size_ok] 96-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:10.449930] I [client.c:2142:notify] 96-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:10.479626] I [client.c:2142:notify] 96-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:10.488424] I [client.c:2142:notify] 96-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:10.497219] I [client.c:2142:notify] 96-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:10.507284] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:10.507403] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:10.507508] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:10.507626] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:10.828378] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:10.829366] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:10.829407] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:10.829475] I [afr-common.c:3627:afr_notify] 95-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:10.829854] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-0: Server lk version = 1
>[2012-06-01 04:27:10.850480] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:10.851392] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:10.851421] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:10.851797] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-1: Server lk version = 1
>[2012-06-01 04:27:10.879712] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:10.880192] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:10.880219] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:10.880286] I [afr-common.c:3627:afr_notify] 95-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:10.880466] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-2: Server lk version = 1
>[2012-06-01 04:27:10.898024] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:10.898283] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:10.898309] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:10.908658] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 95
>[2012-06-01 04:27:10.908743] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-3: Server lk version = 1
>[2012-06-01 04:27:13.064200] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:13.911200] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:13.911537] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:13.911555] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:13.911604] I [afr-common.c:3627:afr_notify] 96-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:13.911678] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-0: Server lk version = 1
>[2012-06-01 04:27:13.926462] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:13.926756] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:13.926776] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:13.927085] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-1: Server lk version = 1
>[2012-06-01 04:27:13.934593] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:13.934830] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:13.934850] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:13.934962] I [afr-common.c:3627:afr_notify] 96-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:13.934992] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-2: Server lk version = 1
>[2012-06-01 04:27:13.947922] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:13.948841] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:13.948861] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:13.978392] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 96
>[2012-06-01 04:27:13.978473] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-3: Server lk version = 1
>[2012-06-01 04:27:14.446814] I [io-cache.c:1549:check_cache_size_ok] 97-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:14.446917] I [io-cache.c:1549:check_cache_size_ok] 97-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:14.446983] I [quota.c:3085:quota_parse_limits] 97-master-quota: no "limit-set" option provided
>[2012-06-01 04:27:14.519172] I [client.c:2142:notify] 97-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:14.553640] I [client.c:2142:notify] 97-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:14.622154] I [client.c:2142:notify] 97-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:14.642572] I [client.c:2142:notify] 97-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:14.661998] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:14.662102] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:14.662288] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:14.662372] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:17.970341] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:17.984057] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:17.984096] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:17.984179] I [afr-common.c:3627:afr_notify] 97-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:17.984645] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-0: Server lk version = 1
>[2012-06-01 04:27:17.984840] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:17.985211] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:17.985241] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:17.995755] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-1: Server lk version = 1
>[2012-06-01 04:27:18.005393] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:18.005911] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:18.005946] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:18.006031] I [afr-common.c:3627:afr_notify] 97-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:18.006111] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:18.006466] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-2: Server lk version = 1
>[2012-06-01 04:27:18.006517] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:18.006539] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:18.023805] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 97
>[2012-06-01 04:27:18.023966] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-3: Server lk version = 1
>[2012-06-01 04:27:28.961159] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:30.153852] I [io-cache.c:1549:check_cache_size_ok] 98-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:30.153947] I [io-cache.c:1549:check_cache_size_ok] 98-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:30.165959] I [client.c:2142:notify] 98-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:30.182862] I [client.c:2142:notify] 98-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:30.196003] I [client.c:2142:notify] 98-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:30.208495] I [client.c:2142:notify] 98-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:30.220031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:30.220146] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:30.220246] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:30.220343] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:33.209789] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:34.001661] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:34.002255] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:34.002275] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:34.002321] I [afr-common.c:3627:afr_notify] 98-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:34.002411] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-0: Server lk version = 1
>[2012-06-01 04:27:34.004908] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:34.005089] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:34.005112] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:34.005340] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-1: Server lk version = 1
>[2012-06-01 04:27:34.013083] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:34.013255] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:34.013279] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:34.013325] I [afr-common.c:3627:afr_notify] 98-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:34.013407] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-2: Server lk version = 1
>[2012-06-01 04:27:34.026577] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:34.026806] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:34.026830] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:34.033315] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 98
>[2012-06-01 04:27:34.033387] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-3: Server lk version = 1
>[2012-06-01 04:27:34.478660] I [io-cache.c:1549:check_cache_size_ok] 99-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:34.478775] I [io-cache.c:1549:check_cache_size_ok] 99-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:34.478867] I [quota.c:3085:quota_parse_limits] 99-master-quota: no "limit-set" option provided
>[2012-06-01 04:27:34.499127] I [client.c:2142:notify] 99-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:34.525230] I [client.c:2142:notify] 99-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:34.541664] I [client.c:2142:notify] 99-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:34.562426] I [client.c:2142:notify] 99-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:34.596011] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:34.596192] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:34.596658] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:34.596800] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:36.930062] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:38.043900] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:38.044134] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:38.044155] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:38.044221] I [afr-common.c:3627:afr_notify] 99-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:38.044249] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-0: Server lk version = 1
>[2012-06-01 04:27:38.047006] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:38.047194] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:38.047213] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:38.047433] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-1: Server lk version = 1
>[2012-06-01 04:27:38.051191] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:38.051371] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:38.051390] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:38.051440] I [afr-common.c:3627:afr_notify] 99-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:38.051522] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-2: Server lk version = 1
>[2012-06-01 04:27:38.054561] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:38.071021] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:38.071046] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:38.075225] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 99
>[2012-06-01 04:27:38.075285] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-3: Server lk version = 1
>[2012-06-01 04:27:43.016941] I [io-cache.c:1549:check_cache_size_ok] 100-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:43.017106] I [io-cache.c:1549:check_cache_size_ok] 100-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:43.117862] I [client.c:2142:notify] 100-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:43.129218] I [client.c:2142:notify] 100-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:43.145764] I [client.c:2142:notify] 100-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:43.162024] I [client.c:2142:notify] 100-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:43.175145] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:43.175287] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:43.175410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:43.175692] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:45.466436] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:46.624182] I [io-cache.c:1549:check_cache_size_ok] 101-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:46.624277] I [io-cache.c:1549:check_cache_size_ok] 101-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:46.624356] I [quota.c:3085:quota_parse_limits] 101-master-quota: no "limit-set" option provided
>[2012-06-01 04:27:46.648730] I [client.c:2142:notify] 101-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:46.659159] I [client.c:2142:notify] 101-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:46.668952] I [client.c:2142:notify] 101-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:46.672486] I [client.c:2142:notify] 101-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:46.684522] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:46.684656] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:46.684763] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:46.685122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:47.066911] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:47.068446] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:47.068498] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:47.068569] I [afr-common.c:3627:afr_notify] 100-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:47.068728] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-0: Server lk version = 1
>[2012-06-01 04:27:47.076853] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:47.077094] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:47.077120] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:47.077499] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-1: Server lk version = 1
>[2012-06-01 04:27:47.086244] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:47.086525] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:47.086549] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:47.086639] I [afr-common.c:3627:afr_notify] 100-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:47.086796] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-2: Server lk version = 1
>[2012-06-01 04:27:47.092551] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:47.093246] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:47.093279] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:47.104061] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 100
>[2012-06-01 04:27:47.104444] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-3: Server lk version = 1
>[2012-06-01 04:27:50.098179] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:50.098486] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:50.098516] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:50.098607] I [afr-common.c:3627:afr_notify] 101-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:50.098745] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-0: Server lk version = 1
>[2012-06-01 04:27:50.104556] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:50.104901] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:50.104932] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:50.105277] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-1: Server lk version = 1
>[2012-06-01 04:27:50.113244] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:50.113540] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:50.113585] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:50.113693] I [afr-common.c:3627:afr_notify] 101-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:50.113836] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-2: Server lk version = 1
>[2012-06-01 04:27:50.124170] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:50.125096] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:50.125132] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:50.130228] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 101
>[2012-06-01 04:27:50.130318] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-3: Server lk version = 1
>[2012-06-01 04:27:54.099641] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:55.156475] I [io-cache.c:1549:check_cache_size_ok] 102-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:55.156907] I [io-cache.c:1549:check_cache_size_ok] 102-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:55.185537] I [client.c:2142:notify] 102-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:55.204068] I [client.c:2142:notify] 102-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:55.211008] I [client.c:2142:notify] 102-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:55.217334] I [client.c:2142:notify] 102-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:55.241210] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:55.241362] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:55.242970] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:55.243112] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:57.540847] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:27:58.712185] I [io-cache.c:1549:check_cache_size_ok] 103-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:27:58.712280] I [io-cache.c:1549:check_cache_size_ok] 103-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:27:58.712352] I [quota.c:3085:quota_parse_limits] 103-master-quota: no "limit-set" option provided
>[2012-06-01 04:27:58.775970] I [client.c:2142:notify] 103-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:58.793424] I [client.c:2142:notify] 103-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:58.806933] I [client.c:2142:notify] 103-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:27:58.824123] I [client.c:2142:notify] 103-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:27:58.850289] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:27:58.850402] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:27:58.850515] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:27:58.850643] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:27:59.129890] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:59.131154] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:27:59.131178] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:59.131234] I [afr-common.c:3627:afr_notify] 102-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:27:59.131330] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-0: Server lk version = 1
>[2012-06-01 04:27:59.138167] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:59.138380] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:27:59.138403] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:59.138705] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-1: Server lk version = 1
>[2012-06-01 04:27:59.143261] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:59.143473] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:27:59.143515] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:59.143575] I [afr-common.c:3627:afr_notify] 102-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:27:59.143700] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-2: Server lk version = 1
>[2012-06-01 04:27:59.147933] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:27:59.148168] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:27:59.148191] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:27:59.155607] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 102
>[2012-06-01 04:27:59.155773] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-3: Server lk version = 1
>[2012-06-01 04:28:01.245889] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:02.160252] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:02.160547] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:02.160569] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:02.160620] I [afr-common.c:3627:afr_notify] 103-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:02.160805] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-0: Server lk version = 1
>[2012-06-01 04:28:02.179281] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:02.179508] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:02.179527] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:02.179660] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-1: Server lk version = 1
>[2012-06-01 04:28:02.195837] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:02.196108] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:02.196128] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:02.196176] I [afr-common.c:3627:afr_notify] 103-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:02.196263] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-2: Server lk version = 1
>[2012-06-01 04:28:02.199509] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:02.199710] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:02.199730] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:02.218380] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 103
>[2012-06-01 04:28:02.218459] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-3: Server lk version = 1
>[2012-06-01 04:28:02.390672] I [io-cache.c:1549:check_cache_size_ok] 104-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:02.390755] I [io-cache.c:1549:check_cache_size_ok] 104-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:02.398676] I [client.c:2142:notify] 104-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:02.411779] I [client.c:2142:notify] 104-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:02.417752] I [client.c:2142:notify] 104-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:02.425577] I [client.c:2142:notify] 104-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:02.435330] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:02.435422] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:02.435504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:02.435593] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:04.655458] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:05.892336] I [io-cache.c:1549:check_cache_size_ok] 105-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:05.892472] I [io-cache.c:1549:check_cache_size_ok] 105-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:05.892567] I [quota.c:3085:quota_parse_limits] 105-master-quota: no "limit-set" option provided
>[2012-06-01 04:28:05.902787] I [client.c:2142:notify] 105-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:05.923724] I [client.c:2142:notify] 105-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:05.938732] I [client.c:2142:notify] 105-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:05.945863] I [client.c:2142:notify] 105-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:05.956878] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:05.957030] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:05.957152] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:05.961959] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:06.224693] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:06.225025] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:06.225054] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:06.225131] I [afr-common.c:3627:afr_notify] 104-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:06.225262] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-0: Server lk version = 1
>[2012-06-01 04:28:06.242495] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:06.242817] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:06.242847] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:06.243144] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-1: Server lk version = 1
>[2012-06-01 04:28:06.252151] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:06.252441] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:06.252484] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:06.252562] I [afr-common.c:3627:afr_notify] 104-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:06.253033] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-2: Server lk version = 1
>[2012-06-01 04:28:06.267060] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:06.267336] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:06.267365] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:06.273478] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 104
>[2012-06-01 04:28:06.273567] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-3: Server lk version = 1
>[2012-06-01 04:28:08.296843] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:09.280315] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:09.281167] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:09.281811] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:09.281930] I [afr-common.c:3627:afr_notify] 105-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:09.282267] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-0: Server lk version = 1
>[2012-06-01 04:28:09.291784] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:09.292978] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:09.293009] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:09.293610] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-1: Server lk version = 1
>[2012-06-01 04:28:09.300516] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:09.300812] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:09.300845] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:09.300945] I [afr-common.c:3627:afr_notify] 105-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:09.301095] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-2: Server lk version = 1
>[2012-06-01 04:28:09.306946] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:09.307224] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:09.307259] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:09.312556] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 105
>[2012-06-01 04:28:09.312741] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-3: Server lk version = 1
>[2012-06-01 04:28:09.378667] I [io-cache.c:1549:check_cache_size_ok] 106-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:09.378794] I [io-cache.c:1549:check_cache_size_ok] 106-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:09.433951] I [client.c:2142:notify] 106-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:09.466883] I [client.c:2142:notify] 106-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:09.480141] I [client.c:2142:notify] 106-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:09.487812] I [client.c:2142:notify] 106-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:09.509123] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:09.513099] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:09.513942] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:09.514063] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:11.855327] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:13.083841] I [io-cache.c:1549:check_cache_size_ok] 107-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:13.083937] I [io-cache.c:1549:check_cache_size_ok] 107-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:13.084001] I [quota.c:3085:quota_parse_limits] 107-master-quota: no "limit-set" option provided
>[2012-06-01 04:28:13.110796] I [client.c:2142:notify] 107-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:13.114013] I [client.c:2142:notify] 107-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:13.125990] I [client.c:2142:notify] 107-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:13.130448] I [client.c:2142:notify] 107-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:13.136989] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:13.137079] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:13.137159] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:13.137311] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:13.319388] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:13.319609] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:13.319644] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:13.319696] I [afr-common.c:3627:afr_notify] 106-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:13.320037] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-0: Server lk version = 1
>[2012-06-01 04:28:13.328978] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:13.329168] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:13.329187] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:13.329366] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-1: Server lk version = 1
>[2012-06-01 04:28:13.339992] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:13.340556] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:13.340580] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:13.340644] I [afr-common.c:3627:afr_notify] 106-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:13.340811] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-2: Server lk version = 1
>[2012-06-01 04:28:13.347885] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:13.348171] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:13.348190] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:13.352233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 106
>[2012-06-01 04:28:13.352303] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-3: Server lk version = 1
>[2012-06-01 04:28:15.386738] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:16.353252] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:16.353578] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:16.353605] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:16.353673] I [afr-common.c:3627:afr_notify] 107-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:16.353796] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-0: Server lk version = 1
>[2012-06-01 04:28:16.366437] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:16.383532] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:16.383563] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:16.383766] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-1: Server lk version = 1
>[2012-06-01 04:28:16.384043] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:16.384139] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:16.384404] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:16.384431] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:16.384511] I [afr-common.c:3627:afr_notify] 107-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:28:16.384641] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:16.384668] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:16.392567] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 107
>[2012-06-01 04:28:16.392655] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-3: Server lk version = 1
>[2012-06-01 04:28:16.392703] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-2: Server lk version = 1
>[2012-06-01 04:28:16.459605] I [io-cache.c:1549:check_cache_size_ok] 108-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:16.459706] I [io-cache.c:1549:check_cache_size_ok] 108-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:16.681414] I [client.c:2142:notify] 108-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:16.712235] I [client.c:2142:notify] 108-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:16.721451] I [client.c:2142:notify] 108-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:16.741437] I [client.c:2142:notify] 108-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:16.752036] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:16.752159] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:16.752268] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:16.752601] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:20.383510] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:20.384184] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:20.384205] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:20.384253] I [afr-common.c:3627:afr_notify] 108-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:20.384336] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-0: Server lk version = 1
>[2012-06-01 04:28:20.390589] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:20.390833] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:20.390854] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:20.391069] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-1: Server lk version = 1
>[2012-06-01 04:28:20.394551] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:20.395365] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:20.395386] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:20.395436] I [afr-common.c:3627:afr_notify] 108-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:20.395571] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-2: Server lk version = 1
>[2012-06-01 04:28:20.404458] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:20.404670] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:20.404691] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:20.418471] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 108
>[2012-06-01 04:28:20.418571] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-3: Server lk version = 1
>[2012-06-01 04:28:28.989496] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:30.322685] I [io-cache.c:1549:check_cache_size_ok] 109-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:30.322797] I [io-cache.c:1549:check_cache_size_ok] 109-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:30.322888] I [quota.c:3085:quota_parse_limits] 109-master-quota: no "limit-set" option provided
>[2012-06-01 04:28:30.364900] I [client.c:2142:notify] 109-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:30.379772] I [client.c:2142:notify] 109-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:30.389501] I [client.c:2142:notify] 109-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:30.398231] I [client.c:2142:notify] 109-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:30.420305] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:30.420406] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:30.420539] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:30.420622] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:33.537234] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:34.421675] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:34.422006] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:34.422036] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:34.422112] I [afr-common.c:3627:afr_notify] 109-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:34.422251] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-0: Server lk version = 1
>[2012-06-01 04:28:34.432943] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:34.433229] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:34.433258] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:34.433331] I [afr-common.c:3627:afr_notify] 109-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:34.433465] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-2: Server lk version = 1
>[2012-06-01 04:28:34.438330] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:34.438635] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:34.438668] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:34.439067] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-3: Server lk version = 1
>[2012-06-01 04:28:34.439777] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:34.440109] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:34.440140] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:34.456686] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 109
>[2012-06-01 04:28:34.456764] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-1: Server lk version = 1
>[2012-06-01 04:28:34.631541] I [io-cache.c:1549:check_cache_size_ok] 110-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:34.633947] I [io-cache.c:1549:check_cache_size_ok] 110-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:34.656885] I [client.c:2142:notify] 110-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:34.675911] I [client.c:2142:notify] 110-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:34.684602] I [client.c:2142:notify] 110-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:34.700030] I [client.c:2142:notify] 110-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:34.721929] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:34.722133] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:34.722218] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:34.722321] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:37.016461] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:38.137220] I [io-cache.c:1549:check_cache_size_ok] 111-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:38.137337] I [io-cache.c:1549:check_cache_size_ok] 111-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:38.137442] I [quota.c:3085:quota_parse_limits] 111-master-quota: no "limit-set" option provided
>[2012-06-01 04:28:38.178533] I [client.c:2142:notify] 111-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:38.190471] I [client.c:2142:notify] 111-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:38.202567] I [client.c:2142:notify] 111-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:38.211659] I [client.c:2142:notify] 111-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:38.226278] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:38.226443] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:38.226579] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:38.226709] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:38.447466] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:38.447814] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:38.447844] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:38.447959] I [afr-common.c:3627:afr_notify] 110-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:38.448108] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-0: Server lk version = 1
>[2012-06-01 04:28:38.455268] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:38.455572] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:38.455604] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:38.455985] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-1: Server lk version = 1
>[2012-06-01 04:28:38.464073] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:38.464362] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:38.464403] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:38.464497] I [afr-common.c:3627:afr_notify] 110-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:38.464640] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-2: Server lk version = 1
>[2012-06-01 04:28:38.471750] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:38.472189] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:38.472223] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:38.481575] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 110
>[2012-06-01 04:28:38.481666] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-3: Server lk version = 1
>[2012-06-01 04:28:41.480297] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:41.480851] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:41.480914] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:41.480998] I [afr-common.c:3627:afr_notify] 111-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:41.481828] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-0: Server lk version = 1
>[2012-06-01 04:28:41.489947] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:41.490271] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:41.490303] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:41.490509] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-1: Server lk version = 1
>[2012-06-01 04:28:41.499368] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:41.499769] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:41.499802] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:41.499920] I [afr-common.c:3627:afr_notify] 111-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:41.500157] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-2: Server lk version = 1
>[2012-06-01 04:28:41.502746] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:41.503694] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:41.503729] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:41.516756] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 111
>[2012-06-01 04:28:41.516857] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-3: Server lk version = 1
>[2012-06-01 04:28:45.444759] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:46.512374] I [io-cache.c:1549:check_cache_size_ok] 112-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:46.512478] I [io-cache.c:1549:check_cache_size_ok] 112-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:46.566693] I [client.c:2142:notify] 112-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:46.576323] I [client.c:2142:notify] 112-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:46.587778] I [client.c:2142:notify] 112-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:46.594735] I [client.c:2142:notify] 112-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:46.621525] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:46.622128] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:46.622677] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:46.622949] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:48.895706] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:50.012830] I [io-cache.c:1549:check_cache_size_ok] 113-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:50.012973] I [io-cache.c:1549:check_cache_size_ok] 113-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:50.013047] I [quota.c:3085:quota_parse_limits] 113-master-quota: no "limit-set" option provided
>[2012-06-01 04:28:50.089850] I [client.c:2142:notify] 113-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:50.105617] I [client.c:2142:notify] 113-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:50.123799] I [client.c:2142:notify] 113-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:50.127304] I [client.c:2142:notify] 113-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:50.147179] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:50.147296] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:50.147464] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:50.147772] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:50.513426] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:50.513812] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:50.513855] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:50.513979] I [afr-common.c:3627:afr_notify] 112-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:50.514027] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-0: Server lk version = 1
>[2012-06-01 04:28:50.522681] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:50.523005] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:50.523037] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:50.523264] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-1: Server lk version = 1
>[2012-06-01 04:28:50.531152] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:50.531470] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:50.531512] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:50.531590] I [afr-common.c:3627:afr_notify] 112-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:50.531665] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-2: Server lk version = 1
>[2012-06-01 04:28:50.537423] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:50.537882] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:50.537921] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:50.550387] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 112
>[2012-06-01 04:28:50.550561] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-3: Server lk version = 1
>[2012-06-01 04:28:52.283472] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:28:53.349563] I [io-cache.c:1549:check_cache_size_ok] 114-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:28:53.362816] I [io-cache.c:1549:check_cache_size_ok] 114-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:28:53.478003] I [client.c:2142:notify] 114-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:53.504183] I [client.c:2142:notify] 114-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:53.517485] I [client.c:2142:notify] 114-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:28:53.533113] I [client.c:2142:notify] 114-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:28:53.555211] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:28:53.555428] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:28:53.555637] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:28:53.555771] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:28:53.563447] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:53.563754] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:53.563786] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:53.563898] I [afr-common.c:3627:afr_notify] 113-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:53.564048] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-0: Server lk version = 1
>[2012-06-01 04:28:53.577959] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:53.578446] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:53.578501] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:53.579036] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-1: Server lk version = 1
>[2012-06-01 04:28:53.587517] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:53.614384] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:53.614457] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:53.614706] I [afr-common.c:3627:afr_notify] 113-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:53.621557] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:53.622073] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-2: Server lk version = 1
>[2012-06-01 04:28:53.624739] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:53.624772] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:53.710682] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 113
>[2012-06-01 04:28:53.710844] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-3: Server lk version = 1
>[2012-06-01 04:28:56.608388] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:56.609205] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:28:56.609237] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:56.609315] I [afr-common.c:3627:afr_notify] 114-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:28:56.609464] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-0: Server lk version = 1
>[2012-06-01 04:28:56.618756] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:56.619294] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:28:56.619329] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:56.619566] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-1: Server lk version = 1
>[2012-06-01 04:28:56.627512] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:56.629564] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:28:56.629598] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:56.629727] I [afr-common.c:3627:afr_notify] 114-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:28:56.629789] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-2: Server lk version = 1
>[2012-06-01 04:28:56.639756] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:28:56.640048] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:28:56.640078] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:28:56.701840] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 114
>[2012-06-01 04:28:56.702086] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-3: Server lk version = 1
>[2012-06-01 04:28:57.783821] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:00.248435] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 114-master-replicate-0: added root inode
>[2012-06-01 04:29:00.249607] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 114-master-replicate-1: added root inode
>[2012-06-01 04:29:00.305831] I [client.c:2151:notify] 58-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:00.311833] I [client.c:2151:notify] 58-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:00.311933] I [client.c:2151:notify] 58-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:00.311988] I [client.c:2151:notify] 58-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:00.311940] I [client.c:2090:client_rpc_notify] 58-master-client-0: disconnected
>[2012-06-01 04:29:00.326726] I [client.c:2090:client_rpc_notify] 58-master-client-1: disconnected
>[2012-06-01 04:29:00.326753] E [afr-common.c:3664:afr_notify] 58-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:29:00.353944] I [client.c:2090:client_rpc_notify] 58-master-client-2: disconnected
>[2012-06-01 04:29:00.354023] I [client.c:2090:client_rpc_notify] 58-master-client-3: disconnected
>[2012-06-01 04:29:00.354048] E [afr-common.c:3664:afr_notify] 58-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:29:00.572980] I [io-cache.c:1549:check_cache_size_ok] 115-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:00.573106] I [io-cache.c:1549:check_cache_size_ok] 115-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:00.573239] I [quota.c:3085:quota_parse_limits] 115-master-quota: no "limit-set" option provided
>[2012-06-01 04:29:00.589808] I [client.c:2142:notify] 115-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:00.596482] I [client.c:2142:notify] 115-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:00.615350] I [client.c:2142:notify] 115-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:00.626706] I [client.c:2142:notify] 115-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:00.661903] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:00.662028] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:00.662105] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:00.662205] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:03.669980] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:03.680114] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:03.684974] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:03.690470] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:03.720654] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:03.720698] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:03.720818] I [afr-common.c:3627:afr_notify] 115-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:03.720905] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:03.720932] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:03.752618] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:03.752662] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:03.752758] I [afr-common.c:3627:afr_notify] 115-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:03.752810] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:03.752832] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:03.772699] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 115
>[2012-06-01 04:29:03.783977] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-2: Server lk version = 1
>[2012-06-01 04:29:03.784021] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-3: Server lk version = 1
>[2012-06-01 04:29:03.784054] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-0: Server lk version = 1
>[2012-06-01 04:29:03.784418] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-1: Server lk version = 1
>[2012-06-01 04:29:10.905512] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:12.678376] I [io-cache.c:1549:check_cache_size_ok] 116-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:12.678498] I [io-cache.c:1549:check_cache_size_ok] 116-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:12.693815] I [client.c:2142:notify] 116-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:12.705993] I [client.c:2142:notify] 116-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:12.715565] I [client.c:2142:notify] 116-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:12.727923] I [client.c:2142:notify] 116-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:12.738339] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:12.738497] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:12.738622] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:12.738764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:16.710905] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:16.711275] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:16.711307] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:16.711390] I [afr-common.c:3627:afr_notify] 116-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:16.711542] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-0: Server lk version = 1
>[2012-06-01 04:29:16.743062] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:16.743383] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:16.743415] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:16.743806] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-1: Server lk version = 1
>[2012-06-01 04:29:16.759581] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:16.760015] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:16.760049] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:16.760156] I [afr-common.c:3627:afr_notify] 116-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:16.760307] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-2: Server lk version = 1
>[2012-06-01 04:29:16.792155] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:16.792541] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:16.792575] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:16.867340] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 116
>[2012-06-01 04:29:16.867458] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-3: Server lk version = 1
>[2012-06-01 04:29:25.842237] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:32.030235] I [io-cache.c:1549:check_cache_size_ok] 117-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:32.030334] I [io-cache.c:1549:check_cache_size_ok] 117-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:32.030425] I [quota.c:3085:quota_parse_limits] 117-master-quota: no "limit-set" option provided
>[2012-06-01 04:29:32.104130] I [client.c:2142:notify] 117-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:32.117570] I [client.c:2142:notify] 117-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:32.136826] I [client.c:2142:notify] 117-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:32.154292] I [client.c:2142:notify] 117-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:32.181353] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:32.181557] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:32.181700] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:32.181809] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:34.488179] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:35.552312] I [io-cache.c:1549:check_cache_size_ok] 118-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:35.552419] I [io-cache.c:1549:check_cache_size_ok] 118-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:35.622708] I [client.c:2142:notify] 118-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:35.643242] I [client.c:2142:notify] 118-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:35.664277] I [client.c:2142:notify] 118-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:35.676976] I [client.c:2142:notify] 118-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:35.691385] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:35.691483] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:35.691580] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:35.691691] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:35.806235] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:35.806518] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:35.806547] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:35.806647] I [afr-common.c:3627:afr_notify] 117-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:35.806774] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-0: Server lk version = 1
>[2012-06-01 04:29:35.848539] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:35.848858] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:35.848885] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:35.849321] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-1: Server lk version = 1
>[2012-06-01 04:29:35.892857] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:35.893123] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:35.893151] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:35.893224] I [afr-common.c:3627:afr_notify] 117-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:35.893352] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-2: Server lk version = 1
>[2012-06-01 04:29:35.930728] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:35.931406] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:35.931438] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:35.949923] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 117
>[2012-06-01 04:29:35.949980] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-3: Server lk version = 1
>[2012-06-01 04:29:37.947125] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:38.951052] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:38.951404] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:38.951435] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:38.951524] I [afr-common.c:3627:afr_notify] 118-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:38.951716] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-0: Server lk version = 1
>[2012-06-01 04:29:38.963022] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:38.963561] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:38.963653] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:38.963780] I [afr-common.c:3627:afr_notify] 118-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:29:38.964004] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-3: Server lk version = 1
>[2012-06-01 04:29:38.990305] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:38.990680] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:38.990710] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:38.991061] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-1: Server lk version = 1
>[2012-06-01 04:29:39.004725] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:39.005286] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:39.005319] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:39.014587] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 118
>[2012-06-01 04:29:39.014686] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-2: Server lk version = 1
>[2012-06-01 04:29:39.100069] I [io-cache.c:1549:check_cache_size_ok] 119-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:39.100206] I [io-cache.c:1549:check_cache_size_ok] 119-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:39.100311] I [quota.c:3085:quota_parse_limits] 119-master-quota: no "limit-set" option provided
>[2012-06-01 04:29:39.108707] I [client.c:2142:notify] 119-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:39.114199] I [client.c:2142:notify] 119-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:39.119646] I [client.c:2142:notify] 119-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:39.124663] I [client.c:2142:notify] 119-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:39.132706] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:39.132891] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:39.133064] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:39.133200] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:39.834918] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 118-master-replicate-0: added root inode
>[2012-06-01 04:29:39.836377] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 118-master-replicate-1: added root inode
>[2012-06-01 04:29:39.907047] I [client.c:2151:notify] 114-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:39.907163] I [client.c:2151:notify] 114-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:39.907210] I [client.c:2151:notify] 114-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:39.907252] I [client.c:2151:notify] 114-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:29:39.907335] I [client.c:2090:client_rpc_notify] 114-master-client-0: disconnected
>[2012-06-01 04:29:39.907381] I [client.c:2090:client_rpc_notify] 114-master-client-1: disconnected
>[2012-06-01 04:29:39.907401] E [afr-common.c:3664:afr_notify] 114-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:29:39.907441] I [client.c:2090:client_rpc_notify] 114-master-client-2: disconnected
>[2012-06-01 04:29:39.907507] I [client.c:2090:client_rpc_notify] 114-master-client-3: disconnected
>[2012-06-01 04:29:39.907529] E [afr-common.c:3664:afr_notify] 114-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:29:41.386477] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:42.432810] I [io-cache.c:1549:check_cache_size_ok] 120-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:42.440015] I [io-cache.c:1549:check_cache_size_ok] 120-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:42.460986] I [client.c:2142:notify] 120-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:42.475183] I [client.c:2142:notify] 120-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:42.486314] I [client.c:2142:notify] 120-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:42.497810] I [client.c:2142:notify] 120-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:42.508436] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:42.510342] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:42.512064] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:42.512199] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:43.025386] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:43.025865] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:43.025915] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:43.026041] I [afr-common.c:3627:afr_notify] 119-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:43.026091] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-0: Server lk version = 1
>[2012-06-01 04:29:43.033171] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:43.033510] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:43.033542] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:43.034006] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-1: Server lk version = 1
>[2012-06-01 04:29:43.042369] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:43.042712] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:43.042745] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:43.042824] I [afr-common.c:3627:afr_notify] 119-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:43.042999] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-2: Server lk version = 1
>[2012-06-01 04:29:43.055119] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:43.055585] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:43.055619] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:43.108452] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 119
>[2012-06-01 04:29:43.108556] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-3: Server lk version = 1
>[2012-06-01 04:29:44.899631] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:46.057710] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:46.058038] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:46.058066] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:46.058161] I [afr-common.c:3627:afr_notify] 120-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:46.058221] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-0: Server lk version = 1
>[2012-06-01 04:29:46.065861] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:46.066753] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:46.066798] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:46.067164] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-1: Server lk version = 1
>[2012-06-01 04:29:46.090084] I [io-cache.c:1549:check_cache_size_ok] 121-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:46.090225] I [io-cache.c:1549:check_cache_size_ok] 121-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:46.090317] I [quota.c:3085:quota_parse_limits] 121-master-quota: no "limit-set" option provided
>[2012-06-01 04:29:46.105877] I [client.c:2142:notify] 121-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:46.120491] I [client.c:2142:notify] 121-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:46.136090] I [client.c:2142:notify] 121-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:46.157560] I [client.c:2142:notify] 121-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:46.178186] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:46.178585] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:46.179054] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:46.179089] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:46.179336] I [afr-common.c:3627:afr_notify] 120-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:46.179491] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-2: Server lk version = 1
>[2012-06-01 04:29:46.179556] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:46.179587] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:46.208130] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 120
>[2012-06-01 04:29:46.208221] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:46.208328] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:46.208409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:46.208502] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-3: Server lk version = 1
>[2012-06-01 04:29:46.208585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:50.108397] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:50.108771] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:50.108804] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:50.108892] I [afr-common.c:3627:afr_notify] 121-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:50.109032] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-0: Server lk version = 1
>[2012-06-01 04:29:50.135885] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:50.136349] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:50.136380] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:50.136986] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-1: Server lk version = 1
>[2012-06-01 04:29:50.147688] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:50.148071] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:50.148104] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:50.148177] I [afr-common.c:3627:afr_notify] 121-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:50.148787] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-2: Server lk version = 1
>[2012-06-01 04:29:50.210320] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:50.211478] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:50.211510] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:50.253177] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 121
>[2012-06-01 04:29:50.253276] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-3: Server lk version = 1
>[2012-06-01 04:29:53.263495] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:54.344286] I [io-cache.c:1549:check_cache_size_ok] 122-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:54.344360] I [io-cache.c:1549:check_cache_size_ok] 122-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:54.380468] I [client.c:2142:notify] 122-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:54.390741] I [client.c:2142:notify] 122-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:54.393604] I [client.c:2142:notify] 122-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:54.401937] I [client.c:2142:notify] 122-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:54.417263] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:54.417331] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:54.417410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:54.417466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:56.768421] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:29:57.937034] I [io-cache.c:1549:check_cache_size_ok] 123-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:29:57.937120] I [io-cache.c:1549:check_cache_size_ok] 123-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:29:57.937186] I [quota.c:3085:quota_parse_limits] 123-master-quota: no "limit-set" option provided
>[2012-06-01 04:29:57.952369] I [client.c:2142:notify] 123-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:57.968731] I [client.c:2142:notify] 123-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:57.978730] I [client.c:2142:notify] 123-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:29:57.985146] I [client.c:2142:notify] 123-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:29:57.997898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:29:57.998002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:29:57.998090] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:29:57.998177] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:29:58.216379] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:58.216716] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:29:58.216739] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:58.216800] I [afr-common.c:3627:afr_notify] 122-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:29:58.217976] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-0: Server lk version = 1
>[2012-06-01 04:29:58.229045] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:58.229313] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:29:58.229336] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:58.231326] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:58.231557] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:29:58.231580] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:58.231662] I [afr-common.c:3627:afr_notify] 122-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:29:58.231766] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-2: Server lk version = 1
>[2012-06-01 04:29:58.232332] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-1: Server lk version = 1
>[2012-06-01 04:29:58.242674] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:29:58.243009] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:29:58.243036] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:29:58.258824] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 122
>[2012-06-01 04:29:58.258887] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-3: Server lk version = 1
>[2012-06-01 04:30:00.604384] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 122-master-replicate-0: added root inode
>[2012-06-01 04:30:00.626505] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 122-master-replicate-1: added root inode
>[2012-06-01 04:30:00.627675] I [client.c:2151:notify] 118-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:30:00.627726] I [client.c:2151:notify] 118-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:30:00.627766] I [client.c:2151:notify] 118-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:30:00.627796] I [client.c:2151:notify] 118-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 04:30:00.631081] I [client.c:2090:client_rpc_notify] 118-master-client-0: disconnected
>[2012-06-01 04:30:00.631120] I [client.c:2090:client_rpc_notify] 118-master-client-1: disconnected
>[2012-06-01 04:30:00.631133] E [afr-common.c:3664:afr_notify] 118-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:30:00.631155] I [client.c:2090:client_rpc_notify] 118-master-client-2: disconnected
>[2012-06-01 04:30:00.631183] I [client.c:2090:client_rpc_notify] 118-master-client-3: disconnected
>[2012-06-01 04:30:00.631194] E [afr-common.c:3664:afr_notify] 118-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 04:30:01.245876] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:01.246090] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:30:01.246109] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:01.246157] I [afr-common.c:3627:afr_notify] 123-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:30:01.246238] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-0: Server lk version = 1
>[2012-06-01 04:30:01.253744] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:01.253977] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:30:01.253997] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:01.254216] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-1: Server lk version = 1
>[2012-06-01 04:30:01.274372] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:01.274810] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:30:01.274836] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:01.274908] I [afr-common.c:3627:afr_notify] 123-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:30:01.275001] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-2: Server lk version = 1
>[2012-06-01 04:30:01.292925] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:01.293153] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:30:01.293175] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:01.299815] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 123
>[2012-06-01 04:30:01.299933] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-3: Server lk version = 1
>[2012-06-01 04:30:05.646089] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:30:06.700797] I [io-cache.c:1549:check_cache_size_ok] 124-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:30:06.700912] I [io-cache.c:1549:check_cache_size_ok] 124-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:30:06.716126] I [client.c:2142:notify] 124-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:06.722277] I [client.c:2142:notify] 124-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:06.725529] I [client.c:2142:notify] 124-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:06.730414] I [client.c:2142:notify] 124-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:30:06.734999] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:30:06.735103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:30:06.735206] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:30:06.735301] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:30:10.304654] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:10.304914] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:30:10.304936] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:10.304989] I [afr-common.c:3627:afr_notify] 124-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:30:10.305109] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-0: Server lk version = 1
>[2012-06-01 04:30:10.309793] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:10.310012] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:30:10.310033] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:10.310221] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-1: Server lk version = 1
>[2012-06-01 04:30:10.317049] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:10.317264] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:30:10.317285] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:10.317335] I [afr-common.c:3627:afr_notify] 124-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:30:10.317425] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-2: Server lk version = 1
>[2012-06-01 04:30:10.323722] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:10.324101] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:30:10.324121] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:10.336215] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 124
>[2012-06-01 04:30:10.336290] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-3: Server lk version = 1
>[2012-06-01 04:30:19.352031] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:30:25.431146] I [io-cache.c:1549:check_cache_size_ok] 125-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:30:25.431273] I [io-cache.c:1549:check_cache_size_ok] 125-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:30:25.431393] I [quota.c:3085:quota_parse_limits] 125-master-quota: no "limit-set" option provided
>[2012-06-01 04:30:25.438730] I [client.c:2142:notify] 125-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:25.445169] I [client.c:2142:notify] 125-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:25.451797] I [client.c:2142:notify] 125-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:25.457021] I [client.c:2142:notify] 125-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:30:25.462742] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:30:25.463013] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:30:25.463088] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:30:25.463173] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:30:27.599222] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:30:28.633723] I [io-cache.c:1549:check_cache_size_ok] 126-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:30:28.633840] I [io-cache.c:1549:check_cache_size_ok] 126-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:30:28.658307] I [client.c:2142:notify] 126-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:28.672352] I [client.c:2142:notify] 126-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:28.680491] I [client.c:2142:notify] 126-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:28.700291] I [client.c:2142:notify] 126-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:30:28.707247] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:30:28.707380] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:30:28.707504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:30:28.707648] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:30:29.346100] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:29.346623] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:30:29.346655] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:29.346766] I [afr-common.c:3627:afr_notify] 125-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-06-01 04:30:29.346942] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-1: Server lk version = 1
>[2012-06-01 04:30:29.351972] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:29.352279] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:30:29.352309] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:29.352718] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-0: Server lk version = 1
>[2012-06-01 04:30:29.358768] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:29.359238] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:30:29.359274] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:29.359377] I [afr-common.c:3627:afr_notify] 125-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-06-01 04:30:29.359423] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-3: Server lk version = 1
>[2012-06-01 04:30:29.364515] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:29.365048] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:30:29.365082] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:29.372940] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 125
>[2012-06-01 04:30:29.373084] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-2: Server lk version = 1
>[2012-06-01 04:30:32.394487] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:32.394753] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:30:32.394773] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:32.394819] I [afr-common.c:3627:afr_notify] 126-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:30:32.394912] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-0: Server lk version = 1
>[2012-06-01 04:30:32.399612] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:32.399913] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:30:32.399934] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:32.400275] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-1: Server lk version = 1
>[2012-06-01 04:30:32.443187] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:32.446198] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:30:32.446232] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:32.446284] I [afr-common.c:3627:afr_notify] 126-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:30:32.446376] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-2: Server lk version = 1
>[2012-06-01 04:30:32.476172] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:32.476483] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:30:32.476514] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:32.493553] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 126
>[2012-06-01 04:30:32.493654] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-3: Server lk version = 1
>[2012-06-01 04:30:41.318300] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2012-06-01 04:30:47.507290] I [io-cache.c:1549:check_cache_size_ok] 127-master-quick-read: Max cache size is 2104233984
>[2012-06-01 04:30:47.507405] I [io-cache.c:1549:check_cache_size_ok] 127-master-io-cache: Max cache size is 2104233984
>[2012-06-01 04:30:47.507504] I [quota.c:3085:quota_parse_limits] 127-master-quota: no "limit-set" option provided
>[2012-06-01 04:30:47.521566] I [client.c:2142:notify] 127-master-client-0: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:47.526308] I [client.c:2142:notify] 127-master-client-1: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:47.530910] I [client.c:2142:notify] 127-master-client-2: parent translators are ready, attempting connect on transport
>[2012-06-01 04:30:47.536805] I [client.c:2142:notify] 127-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.151
>  4:     option remote-subvolume /exportdir/d1
>  5:     option transport-type tcp
>  6:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
>  7:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.151
> 13:     option remote-subvolume /exportdir/d2
> 14:     option transport-type tcp
> 15:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 16:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.151
> 22:     option remote-subvolume /exportdir/d3
> 23:     option transport-type tcp
> 24:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 25:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.151
> 31:     option remote-subvolume /exportdir/d4
> 32:     option transport-type tcp
> 33:     option username ab033ab9-6b51-4547-a7b5-41c06850a6e7
> 34:     option password cb4e4265-d329-4ee5-85d3-ae6c76565f06
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-quota
> 53:     type features/quota
> 54:     option timeout 0
> 55:     subvolumes master-dht
> 56: end-volume
> 57: 
> 58: volume master-write-behind
> 59:     type performance/write-behind
> 60:     subvolumes master-quota
> 61: end-volume
> 62: 
> 63: volume master-read-ahead
> 64:     type performance/read-ahead
> 65:     subvolumes master-write-behind
> 66: end-volume
> 67: 
> 68: volume master-io-cache
> 69:     type performance/io-cache
> 70:     subvolumes master-read-ahead
> 71: end-volume
> 72: 
> 73: volume master-quick-read
> 74:     type performance/quick-read
> 75:     subvolumes master-io-cache
> 76: end-volume
> 77: 
> 78: volume master-md-cache
> 79:     type performance/md-cache
> 80:     subvolumes master-quick-read
> 81: end-volume
> 82: 
> 83: volume master
> 84:     type debug/io-stats
> 85:     option latency-measurement off
> 86:     option count-fop-hits off
> 87:     subvolumes master-md-cache
> 88: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-06-01 04:30:47.546250] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-0: changing port to 24009 (from 0)
>[2012-06-01 04:30:47.546383] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-1: changing port to 24010 (from 0)
>[2012-06-01 04:30:47.546504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-2: changing port to 24011 (from 0)
>[2012-06-01 04:30:47.546661] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-3: changing port to 24012 (from 0)
>[2012-06-01 04:30:51.497964] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:51.498317] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'.
>[2012-06-01 04:30:51.498346] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:51.498459] I [afr-common.c:3627:afr_notify] 127-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-06-01 04:30:51.498506] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-0: Server lk version = 1
>[2012-06-01 04:30:51.504268] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:51.504603] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'.
>[2012-06-01 04:30:51.504631] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:51.504817] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-1: Server lk version = 1
>[2012-06-01 04:30:51.509815] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:51.510520] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'.
>[2012-06-01 04:30:51.510550] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:51.510645] I [afr-common.c:3627:afr_notify] 127-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-06-01 04:30:51.510801] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-2: Server lk version = 1
>[2012-06-01 04:30:51.517811] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330)
>[2012-06-01 04:30:51.518437] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'.
>[2012-06-01 04:30:51.518469] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-06-01 04:30:51.524304] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 127
>[2012-06-01 04:30:51.524954] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-3: Server lk version = 1
>[2012-06-01 05:00:29.938055] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 127-master-replicate-0: added root inode
>[2012-06-01 05:00:29.941310] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 127-master-replicate-1: added root inode
>[2012-06-01 05:00:29.944795] I [client.c:2151:notify] 122-master-client-0: current graph is no longer active, destroying rpc_client 
>[2012-06-01 05:00:29.944908] I [client.c:2151:notify] 122-master-client-1: current graph is no longer active, destroying rpc_client 
>[2012-06-01 05:00:29.945175] I [client.c:2090:client_rpc_notify] 122-master-client-0: disconnected
>[2012-06-01 05:00:29.945190] I [client.c:2151:notify] 122-master-client-2: current graph is no longer active, destroying rpc_client 
>[2012-06-01 05:00:29.945234] I [client.c:2090:client_rpc_notify] 122-master-client-1: disconnected
>[2012-06-01 05:00:29.945257] E [afr-common.c:3664:afr_notify] 122-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2012-06-01 05:00:29.945304] I [client.c:2151:notify] 122-master-client-3: current graph is no longer active, destroying rpc_client 
>[2012-06-01 05:00:29.945523] I [client.c:2090:client_rpc_notify] 122-master-client-2: disconnected
>[2012-06-01 05:00:29.945892] I [client.c:2090:client_rpc_notify] 122-master-client-3: disconnected
>[2012-06-01 05:00:29.945920] E [afr-common.c:3664:afr_notify] 122-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.

Actions: View

Attachments on bug 827405: 588419