Login
Log in using an SSO provider:
Fedora Account System
Red Hat Associate
Red Hat Customer
Login using a Red Hat Bugzilla account
Forgot Password
Create an Account
Red Hat Bugzilla – Attachment 588419 Details for
Bug 827405
Dbench exit with "Bad file descriptor" if the graph-change happens
Home
New
Search
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh92 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
[?]
This site requires JavaScript to be enabled to function correctly, please enable it.
Client log file.
root-client.log (text/x-log), 1005.78 KB, created by
Vijaykumar Koppad
on 2012-06-01 11:37:49 UTC
(
hide
)
Description:
Client log file.
Filename:
MIME Type:
Creator:
Vijaykumar Koppad
Created:
2012-06-01 11:37:49 UTC
Size:
1005.78 KB
patch
obsolete
>[2012-06-01 03:57:44.687411] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa45 >[2012-06-01 03:57:44.707685] E [glusterfsd-mgmt.c:1555:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server >[2012-06-01 03:57:44.707776] E [glusterfsd-mgmt.c:1623:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:doa) >[2012-06-01 03:57:44.708020] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb6) [0x3b4180f516] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x3b4180f302] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x298) [0x40c348]))) 0-: received signum (0), shutting down >[2012-06-01 03:57:44.708132] I [fuse-bridge.c:4643:fini] 0-fuse: Unmounting '/root/client'. >[2012-06-01 03:57:58.155693] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa45 >[2012-06-01 03:57:58.173405] I [io-cache.c:1549:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104233984 >[2012-06-01 03:57:58.173555] I [io-cache.c:1549:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104233984 >[2012-06-01 03:57:58.194193] I [client.c:2142:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 03:57:58.199347] I [client.c:2142:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 03:57:58.204278] I [client.c:2142:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 03:57:58.209216] I [client.c:2142:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 03:57:58.215571] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-06-01 03:57:58.215712] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-06-01 03:57:58.215844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-06-01 03:57:58.216022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-06-01 03:58:02.169524] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 03:58:02.169947] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 03:58:02.169981] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 03:58:02.170061] I [afr-common.c:3627:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 03:58:02.170212] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-06-01 03:58:02.174970] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 03:58:02.175285] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 03:58:02.175316] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 03:58:02.176605] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-06-01 03:58:02.179798] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 03:58:02.180102] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 03:58:02.180131] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 03:58:02.180203] I [afr-common.c:3627:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 03:58:02.180336] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-06-01 03:58:02.184607] I [client-handshake.c:1636:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 03:58:02.185052] I [client-handshake.c:1433:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 03:58:02.185083] I [client-handshake.c:1445:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 03:58:02.199890] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 0 >[2012-06-01 03:58:02.200123] I [client-handshake.c:453:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-06-01 03:58:02.200396] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 >[2012-06-01 03:58:02.201091] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-06-01 03:58:02.201849] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-06-01 04:00:02.498962] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:00:05.108168] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2012-06-01 04:16:46.779298] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:16:47.899066] I [io-cache.c:1549:check_cache_size_ok] 1-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:16:47.899189] I [io-cache.c:1549:check_cache_size_ok] 1-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:16:47.899290] I [quota.c:3085:quota_parse_limits] 1-master-quota: no "limit-set" option provided >[2012-06-01 04:16:47.977016] I [client.c:2142:notify] 1-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:47.992436] I [client.c:2142:notify] 1-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:48.011814] I [client.c:2142:notify] 1-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:48.016562] I [client.c:2142:notify] 1-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:16:48.031955] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:16:48.032062] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:16:48.032147] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:16:48.032230] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 1-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:16:49.993769] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:16:51.027484] I [io-cache.c:1549:check_cache_size_ok] 2-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:16:51.027600] I [io-cache.c:1549:check_cache_size_ok] 2-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:16:51.038352] I [client.c:2142:notify] 2-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:51.046975] I [client.c:2142:notify] 2-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:51.063556] I [client.c:2142:notify] 2-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:51.077667] I [client.c:2142:notify] 2-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:16:51.089880] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:16:51.090349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:16:51.090852] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:16:51.090981] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 2-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:16:51.371313] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:51.371763] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:16:51.371796] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:51.371919] I [afr-common.c:3627:afr_notify] 1-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:16:51.372076] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-0: Server lk version = 1 >[2012-06-01 04:16:51.376478] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:51.376822] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:16:51.376854] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:51.377262] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-1: Server lk version = 1 >[2012-06-01 04:16:51.382279] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:51.382709] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:16:51.382747] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:51.382853] I [afr-common.c:3627:afr_notify] 1-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:16:51.382938] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-2: Server lk version = 1 >[2012-06-01 04:16:51.387490] I [client-handshake.c:1636:select_server_supported_programs] 1-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:51.387905] I [client-handshake.c:1433:client_setvolume_cbk] 1-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:16:51.387937] I [client-handshake.c:1445:client_setvolume_cbk] 1-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:51.399939] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 1 >[2012-06-01 04:16:51.400060] I [client-handshake.c:453:client_set_lk_version_cbk] 1-master-client-3: Server lk version = 1 >[2012-06-01 04:16:53.218357] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:16:54.329266] I [io-cache.c:1549:check_cache_size_ok] 3-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:16:54.329390] I [io-cache.c:1549:check_cache_size_ok] 3-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:16:54.329505] I [quota.c:3085:quota_parse_limits] 3-master-quota: no "limit-set" option provided >[2012-06-01 04:16:54.373906] I [client.c:2142:notify] 3-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:54.383603] I [client.c:2142:notify] 3-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:54.431894] I [client.c:2142:notify] 3-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:54.455705] I [client.c:2142:notify] 3-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:16:54.483504] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:54.484842] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:54.486479] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:54.487573] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:16:54.487829] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:16:54.487892] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:54.490010] I [afr-common.c:3627:afr_notify] 2-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:16:54.490074] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:16:54.490210] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:16:54.490240] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:54.492163] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:16:54.492417] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:16:54.492448] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:54.492658] I [afr-common.c:3627:afr_notify] 2-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:16:54.492711] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 3-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:16:54.492836] I [client-handshake.c:1636:select_server_supported_programs] 2-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:54.493145] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-0: Server lk version = 1 >[2012-06-01 04:16:54.493204] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-1: Server lk version = 1 >[2012-06-01 04:16:54.493253] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-2: Server lk version = 1 >[2012-06-01 04:16:54.493312] I [client-handshake.c:1433:client_setvolume_cbk] 2-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:16:54.493336] I [client-handshake.c:1445:client_setvolume_cbk] 2-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:54.506279] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 2 >[2012-06-01 04:16:54.507484] I [client-handshake.c:453:client_set_lk_version_cbk] 2-master-client-3: Server lk version = 1 >[2012-06-01 04:16:56.437676] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:16:57.469105] I [io-cache.c:1549:check_cache_size_ok] 4-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:16:57.469260] I [io-cache.c:1549:check_cache_size_ok] 4-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:16:57.491542] I [client.c:2142:notify] 4-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:57.499254] I [client.c:2142:notify] 4-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:57.516879] I [client.c:2142:notify] 4-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:16:57.524054] I [client.c:2142:notify] 4-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:16:57.532017] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:16:57.532263] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:16:57.532393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:16:57.532587] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 4-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:16:58.488513] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:58.488842] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:16:58.488904] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:58.488988] I [afr-common.c:3627:afr_notify] 3-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:16:58.489130] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-0: Server lk version = 1 >[2012-06-01 04:16:58.495933] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:58.496233] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:16:58.496264] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:58.496614] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-1: Server lk version = 1 >[2012-06-01 04:16:58.504902] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:58.505251] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:16:58.505284] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:58.505378] I [afr-common.c:3627:afr_notify] 3-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:16:58.505463] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-2: Server lk version = 1 >[2012-06-01 04:16:58.511818] I [client-handshake.c:1636:select_server_supported_programs] 3-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:16:58.512133] I [client-handshake.c:1433:client_setvolume_cbk] 3-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:16:58.512163] I [client-handshake.c:1445:client_setvolume_cbk] 3-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:16:58.520610] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 3 >[2012-06-01 04:16:58.520700] I [client-handshake.c:453:client_set_lk_version_cbk] 3-master-client-3: Server lk version = 1 >[2012-06-01 04:16:59.671090] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:16:59.726935] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 3-master-replicate-0: added root inode >[2012-06-01 04:16:59.728193] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 3-master-replicate-1: added root inode >[2012-06-01 04:16:59.728473] I [client.c:2151:notify] 0-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:16:59.728769] I [client.c:2090:client_rpc_notify] 0-master-client-0: disconnected >[2012-06-01 04:16:59.728807] I [client.c:2151:notify] 0-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:16:59.729017] I [client.c:2090:client_rpc_notify] 0-master-client-1: disconnected >[2012-06-01 04:16:59.729044] E [afr-common.c:3664:afr_notify] 0-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:16:59.729071] I [client.c:2151:notify] 0-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:16:59.729331] I [client.c:2090:client_rpc_notify] 0-master-client-2: disconnected >[2012-06-01 04:16:59.729370] I [client.c:2151:notify] 0-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:16:59.729594] I [client.c:2090:client_rpc_notify] 0-master-client-3: disconnected >[2012-06-01 04:16:59.729650] E [afr-common.c:3664:afr_notify] 0-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:01.518092] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:01.518261] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:01.518421] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:01.518443] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:01.518567] I [afr-common.c:3627:afr_notify] 4-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:01.518601] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:01.518633] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:01.518897] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-0: Server lk version = 1 >[2012-06-01 04:17:01.518922] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-1: Server lk version = 1 >[2012-06-01 04:17:01.535050] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:01.535289] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:01.535309] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:01.535355] I [afr-common.c:3627:afr_notify] 4-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:17:01.535561] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-2: Server lk version = 1 >[2012-06-01 04:17:01.547135] I [client-handshake.c:1636:select_server_supported_programs] 4-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:01.547300] I [client-handshake.c:1433:client_setvolume_cbk] 4-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:01.547318] I [client-handshake.c:1445:client_setvolume_cbk] 4-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:01.555111] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 4 >[2012-06-01 04:17:01.556083] I [client-handshake.c:453:client_set_lk_version_cbk] 4-master-client-3: Server lk version = 1 >[2012-06-01 04:17:01.557738] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 4-master-replicate-0: added root inode >[2012-06-01 04:17:01.559243] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 4-master-replicate-1: added root inode >[2012-06-01 04:17:01.561167] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8522070e-0759-48a4-897b-8b3018f587e0) >[2012-06-01 04:17:01.563676] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4e9ef3d5-fab8-4169-80b7-e6cbc2d798a5) >[2012-06-01 04:17:01.566271] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:17c95f95-37e1-4ae1-898f-f64d4d4f0bbc) >[2012-06-01 04:17:01.568284] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:de0af784-3a82-4504-bb61-3a2e4880f172) >[2012-06-01 04:17:01.572632] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e22b122f-a193-4fae-83f4-14c7f805e1c7) >[2012-06-01 04:17:01.575349] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9877cbfe-cf9b-4d11-8070-0d5eec833f9b) >[2012-06-01 04:17:01.578231] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:2796ef76-229d-4d3a-aceb-310a82684fdb) >[2012-06-01 04:17:01.580389] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d9a23130-a7c3-431e-8398-282a8b4a083a) >[2012-06-01 04:17:01.583641] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:5f5cc251-41a9-4d27-a77a-3a8cb8fabbdf) >[2012-06-01 04:17:01.586505] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f2031c96-8c98-4c06-8963-7f980665e88a) >[2012-06-01 04:17:01.590049] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:04649108-a2c6-48d1-a62f-7847350d311f) >[2012-06-01 04:17:01.592927] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:3e92e90c-9959-46fa-87a3-f29e1cb32158) >[2012-06-01 04:17:01.595687] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:7d49129a-acb5-4b81-b56a-73532c46a36b) >[2012-06-01 04:17:01.598705] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:962d7a36-4f82-4dd3-9fc7-92be4ef55549) >[2012-06-01 04:17:01.601817] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:032921e9-9e6f-4468-b48b-f67cc8d1c4c7) >[2012-06-01 04:17:01.604429] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:478c58a9-a0d2-4920-94c3-845aef639a4e) >[2012-06-01 04:17:01.607739] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b57e5fb2-1b88-47c5-bba7-9a8fd56eae4f) >[2012-06-01 04:17:01.610596] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:ba221d26-f588-4f8d-af77-c8940b63e797) >[2012-06-01 04:17:01.613661] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f3910016-2214-42fc-a9b8-7cf3b2c896b8) >[2012-06-01 04:17:01.617380] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:044a8cdc-2180-459f-b048-f1049910ffc2) >[2012-06-01 04:17:01.619785] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f428c0bc-6194-4cd3-956d-676d323429e8) >[2012-06-01 04:17:01.621947] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a4b59479-af1f-4a69-9a5d-116a49047b7b) >[2012-06-01 04:17:01.624524] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:bba7d9dd-596b-48d4-ae5f-e71a91d7e34f) >[2012-06-01 04:17:01.627282] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8ad59a9b-7b83-4ef0-a2b5-ec0df7616c1d) >[2012-06-01 04:17:01.629543] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9e50920b-78b8-49aa-8889-7976dbc3ea80) >[2012-06-01 04:17:01.632500] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4f58cedd-dd70-406d-a818-a99ed956b248) >[2012-06-01 04:17:01.635978] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:563e9bfd-8c7e-4dbd-8657-85115a5ce75c) >[2012-06-01 04:17:01.638649] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:c2bd75ce-c798-4a34-9f60-e32b9ef3b250) >[2012-06-01 04:17:01.642217] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:32e09d06-6e72-458d-81da-0befb2e7e160) >[2012-06-01 04:17:01.644989] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:561277dc-03e2-4127-baee-b556fc9799f3) >[2012-06-01 04:17:01.648016] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e30db81a-ae6d-449b-a1c1-ba228ddf1da3) >[2012-06-01 04:17:01.651338] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0e1cf4e7-4ec0-4550-8d0f-c94046418fc4) >[2012-06-01 04:17:01.654320] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d3f0ce18-5bf7-4b93-a3c4-3caf801b20de) >[2012-06-01 04:17:01.657209] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b77136c4-6e01-4db2-b5d5-dbf780d29375) >[2012-06-01 04:17:01.659886] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:efbd2ddc-037a-4951-a85f-5de8826810a0) >[2012-06-01 04:17:01.662492] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8085d7f1-27c8-4cb3-8543-e874db6331b1) >[2012-06-01 04:17:01.665157] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8e70a386-2f45-4bef-be5f-c6569a0a0a49) >[2012-06-01 04:17:01.667787] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:cd07e6d5-73d1-4a16-a904-21c930e98701) >[2012-06-01 04:17:01.670779] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d7746c28-3d8b-4ae5-bef0-544df2868b25) >[2012-06-01 04:17:01.673147] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:5b58fab9-d2fb-4492-9ff0-e0d627cd9fa0) >[2012-06-01 04:17:01.675505] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e368c3c4-9b39-45c6-89a8-aa019c499002) >[2012-06-01 04:17:01.678379] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:becfff3d-933c-43bb-a7a9-ec4f76041138) >[2012-06-01 04:17:01.680715] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:3da1b304-76f6-4573-945c-e708c6642e66) >[2012-06-01 04:17:01.683306] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a9b8a98e-ef6d-4b0f-be31-fca72cb624b5) >[2012-06-01 04:17:01.686571] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:15d64ae4-84d6-47ed-a370-33ee91dd7f92) >[2012-06-01 04:17:01.689232] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:d55db600-df01-40a6-852b-f66cd5332de7) >[2012-06-01 04:17:01.692181] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:2431005a-eb7c-4f4b-a371-a2aad1955d87) >[2012-06-01 04:17:01.694919] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:1f0811fa-61fa-44a3-ab10-f3056af73053) >[2012-06-01 04:17:01.698076] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9ee41d89-5c9c-4e7f-b520-f08c1f300ffb) >[2012-06-01 04:17:01.701090] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:35b797aa-872a-45e0-992f-6c0a67000ff3) >[2012-06-01 04:17:01.704066] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:54f59351-8320-4797-9c8b-a4a331898cbf) >[2012-06-01 04:17:01.706750] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:9e73e512-c3c1-444a-89a4-97a067332105) >[2012-06-01 04:17:01.711075] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4c355011-989c-45a2-9457-833c6e46e40f) >[2012-06-01 04:17:01.717953] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:58397717-8352-4bf1-b139-98b8ede2f7bb) >[2012-06-01 04:17:01.720571] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:8dbd1880-d485-4e32-a7cc-1a31b75cc066) >[2012-06-01 04:17:01.723412] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:4de155a8-c85c-4731-953f-60a6ff185495) >[2012-06-01 04:17:01.725901] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:a205824e-0f70-44a8-92f1-866792d9b745) >[2012-06-01 04:17:01.728428] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:f9643bbd-82db-467d-9c81-7980608f1f0e) >[2012-06-01 04:17:01.731076] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e8aa16cd-95b3-44bd-a0b0-be8ddab1090f) >[2012-06-01 04:17:01.733462] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:cf20a194-40aa-48a1-8269-c404d78ab3a3) >[2012-06-01 04:17:01.737083] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:24f4f8d4-1d0d-4530-b8a3-bfc24aecc4b0) >[2012-06-01 04:17:01.739368] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:95ac252d-f0af-4e81-b603-3f874fccdfb7) >[2012-06-01 04:17:01.742035] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e02daf6d-11be-49a9-92b4-0efc44ae6c6c) >[2012-06-01 04:17:01.744525] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:23843325-21f3-4669-82c5-fa30c1eee6fb) >[2012-06-01 04:17:01.746875] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0f4ae84c-8e56-4f09-a397-87a7d210cc75) >[2012-06-01 04:17:01.749312] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:6edeed61-f4ea-420c-b32d-dc461e950f58) >[2012-06-01 04:17:01.751652] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:e42acc3f-d1b0-4377-afa2-612fc4559d33) >[2012-06-01 04:17:01.753860] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:b6bdf08d-87e0-48b6-a675-ca45022bed58) >[2012-06-01 04:17:01.756288] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:23b92615-8b00-4f7d-919b-dd30d836b438) >[2012-06-01 04:17:01.759792] W [quota.c:2377:quota_fsync_cbk] 3-master-quota: quota context not set in inode (gfid:0b979563-23c4-4715-a48c-3b22ae6aa32e) >[2012-06-01 04:17:01.761458] I [client.c:2151:notify] 3-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:01.761499] I [client.c:2151:notify] 3-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:01.761535] I [client.c:2151:notify] 3-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:01.761561] I [client.c:2151:notify] 3-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:01.771447] I [client.c:2090:client_rpc_notify] 3-master-client-0: disconnected >[2012-06-01 04:17:01.771491] I [client.c:2090:client_rpc_notify] 3-master-client-1: disconnected >[2012-06-01 04:17:01.771503] E [afr-common.c:3664:afr_notify] 3-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:01.771529] I [client.c:2090:client_rpc_notify] 3-master-client-2: disconnected >[2012-06-01 04:17:01.771557] I [client.c:2090:client_rpc_notify] 3-master-client-3: disconnected >[2012-06-01 04:17:01.771568] E [afr-common.c:3664:afr_notify] 3-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:10.793972] I [io-cache.c:1549:check_cache_size_ok] 5-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:10.794083] I [io-cache.c:1549:check_cache_size_ok] 5-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:10.794146] I [quota.c:3085:quota_parse_limits] 5-master-quota: no "limit-set" option provided >[2012-06-01 04:17:10.807828] I [client.c:2142:notify] 5-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:10.821847] I [client.c:2142:notify] 5-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:10.826423] I [client.c:2142:notify] 5-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:10.839797] I [client.c:2142:notify] 5-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:10.848932] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:10.849037] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:10.849652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:10.849751] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 5-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:17:14.561062] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:14.561660] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:14.561691] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:14.561871] I [afr-common.c:3627:afr_notify] 5-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:14.563211] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-0: Server lk version = 1 >[2012-06-01 04:17:14.567154] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:14.567456] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:14.567486] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:14.568141] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-1: Server lk version = 1 >[2012-06-01 04:17:14.578330] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:14.579779] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:14.579813] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:14.579912] I [afr-common.c:3627:afr_notify] 5-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:17:14.581027] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-2: Server lk version = 1 >[2012-06-01 04:17:14.581074] I [client-handshake.c:1636:select_server_supported_programs] 5-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:14.581921] I [client-handshake.c:1433:client_setvolume_cbk] 5-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:14.581954] I [client-handshake.c:1445:client_setvolume_cbk] 5-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:14.587230] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 5 >[2012-06-01 04:17:14.588460] I [client-handshake.c:453:client_set_lk_version_cbk] 5-master-client-3: Server lk version = 1 >[2012-06-01 04:17:14.589303] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 5-master-replicate-0: added root inode >[2012-06-01 04:17:14.590543] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 5-master-replicate-1: added root inode >[2012-06-01 04:17:14.629461] I [client.c:2151:notify] 4-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:14.629549] I [client.c:2151:notify] 4-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:14.629598] I [client.c:2151:notify] 4-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:14.629671] I [client.c:2151:notify] 4-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:14.633823] I [client.c:2090:client_rpc_notify] 4-master-client-0: disconnected >[2012-06-01 04:17:14.633926] I [client.c:2090:client_rpc_notify] 4-master-client-1: disconnected >[2012-06-01 04:17:14.633948] E [afr-common.c:3664:afr_notify] 4-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:14.633982] I [client.c:2090:client_rpc_notify] 4-master-client-2: disconnected >[2012-06-01 04:17:14.634019] I [client.c:2090:client_rpc_notify] 4-master-client-3: disconnected >[2012-06-01 04:17:14.634037] E [afr-common.c:3664:afr_notify] 4-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:18.007973] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:17:29.070840] I [io-cache.c:1549:check_cache_size_ok] 6-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:29.070935] I [io-cache.c:1549:check_cache_size_ok] 6-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:29.083663] I [client.c:2142:notify] 6-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:29.091641] I [client.c:2142:notify] 6-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:29.097010] I [client.c:2142:notify] 6-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:29.101539] I [client.c:2142:notify] 6-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:29.110780] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:29.110923] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:29.111006] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:29.111107] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 6-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:17:31.338187] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:17:32.477787] I [io-cache.c:1549:check_cache_size_ok] 7-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:32.477874] I [io-cache.c:1549:check_cache_size_ok] 7-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:32.477933] I [quota.c:3085:quota_parse_limits] 7-master-quota: no "limit-set" option provided >[2012-06-01 04:17:32.665235] I [client.c:2142:notify] 7-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:32.674794] I [client.c:2142:notify] 7-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:32.683919] I [client.c:2142:notify] 7-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:32.686893] I [client.c:2142:notify] 7-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:32.695306] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:32.695691] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:32.701682] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:32.701948] I [client-handshake.c:1636:select_server_supported_programs] 6-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:32.703247] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:32.703269] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:32.703403] I [afr-common.c:3627:afr_notify] 6-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:32.703444] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:32.703457] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:32.703815] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:32.703839] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:32.703966] I [afr-common.c:3627:afr_notify] 6-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:17:32.704002] I [client-handshake.c:1433:client_setvolume_cbk] 6-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:32.704016] I [client-handshake.c:1445:client_setvolume_cbk] 6-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:32.746000] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 6 >[2012-06-01 04:17:32.746103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:32.746624] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-0: Server lk version = 1 >[2012-06-01 04:17:32.746713] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-1: Server lk version = 1 >[2012-06-01 04:17:32.746739] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-2: Server lk version = 1 >[2012-06-01 04:17:32.746759] I [client-handshake.c:453:client_set_lk_version_cbk] 6-master-client-3: Server lk version = 1 >[2012-06-01 04:17:32.746795] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:17:32.746945] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:32.747029] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 7-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:32.750812] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 6-master-replicate-0: added root inode >[2012-06-01 04:17:32.752575] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 6-master-replicate-1: added root inode >[2012-06-01 04:17:32.769110] I [client.c:2151:notify] 5-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:32.769165] I [client.c:2151:notify] 5-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:32.769194] I [client.c:2151:notify] 5-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:32.769224] I [client.c:2151:notify] 5-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:32.771163] I [client.c:2090:client_rpc_notify] 5-master-client-0: disconnected >[2012-06-01 04:17:32.771359] I [client.c:2090:client_rpc_notify] 5-master-client-1: disconnected >[2012-06-01 04:17:32.771376] E [afr-common.c:3664:afr_notify] 5-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:32.771399] I [client.c:2090:client_rpc_notify] 5-master-client-2: disconnected >[2012-06-01 04:17:32.771421] I [client.c:2090:client_rpc_notify] 5-master-client-3: disconnected >[2012-06-01 04:17:32.771432] E [afr-common.c:3664:afr_notify] 5-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:34.891133] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:17:35.937661] I [io-cache.c:1549:check_cache_size_ok] 8-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:35.937780] I [io-cache.c:1549:check_cache_size_ok] 8-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:35.945468] I [client.c:2142:notify] 8-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:35.952804] I [client.c:2142:notify] 8-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:35.957686] I [client.c:2142:notify] 8-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:35.962700] I [client.c:2142:notify] 8-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:35.974803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:35.974984] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:35.975112] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:35.975236] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 8-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:17:36.686607] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:36.694161] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:36.694674] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:36.694901] I [afr-common.c:3627:afr_notify] 7-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:36.700605] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-0: Server lk version = 1 >[2012-06-01 04:17:36.714261] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:36.714555] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:36.716469] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:36.716503] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:36.719223] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-1: Server lk version = 1 >[2012-06-01 04:17:36.720271] I [client-handshake.c:1636:select_server_supported_programs] 7-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:36.720596] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:36.720782] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:36.721306] I [afr-common.c:3627:afr_notify] 7-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:17:36.723026] I [client-handshake.c:1433:client_setvolume_cbk] 7-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:36.723063] I [client-handshake.c:1445:client_setvolume_cbk] 7-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:36.733079] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 7 >[2012-06-01 04:17:36.733175] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-3: Server lk version = 1 >[2012-06-01 04:17:36.734654] I [client-handshake.c:453:client_set_lk_version_cbk] 7-master-client-2: Server lk version = 1 >[2012-06-01 04:17:36.738713] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 7-master-replicate-0: added root inode >[2012-06-01 04:17:36.741365] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 7-master-replicate-1: added root inode >[2012-06-01 04:17:36.807314] I [client.c:2151:notify] 6-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:36.807426] I [client.c:2151:notify] 6-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:36.807478] I [client.c:2151:notify] 6-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:36.807522] I [client.c:2151:notify] 6-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:36.807524] I [client.c:2090:client_rpc_notify] 6-master-client-0: disconnected >[2012-06-01 04:17:36.807633] I [client.c:2090:client_rpc_notify] 6-master-client-1: disconnected >[2012-06-01 04:17:36.807656] E [afr-common.c:3664:afr_notify] 6-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:36.807689] I [client.c:2090:client_rpc_notify] 6-master-client-2: disconnected >[2012-06-01 04:17:36.807722] I [client.c:2090:client_rpc_notify] 6-master-client-3: disconnected >[2012-06-01 04:17:36.807757] E [afr-common.c:3664:afr_notify] 6-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:38.141204] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:17:39.718927] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:39.719263] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:39.719283] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:39.719417] I [afr-common.c:3627:afr_notify] 8-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:39.719730] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-0: Server lk version = 1 >[2012-06-01 04:17:39.723966] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:39.725516] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:39.725537] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:39.726996] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-1: Server lk version = 1 >[2012-06-01 04:17:39.730607] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:39.733368] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:39.733625] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:39.733742] I [afr-common.c:3627:afr_notify] 8-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:17:39.734987] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-2: Server lk version = 1 >[2012-06-01 04:17:39.741719] I [client-handshake.c:1636:select_server_supported_programs] 8-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:39.742249] I [client-handshake.c:1433:client_setvolume_cbk] 8-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:39.742269] I [client-handshake.c:1445:client_setvolume_cbk] 8-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:39.748880] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 8 >[2012-06-01 04:17:39.749985] I [client-handshake.c:453:client_set_lk_version_cbk] 8-master-client-3: Server lk version = 1 >[2012-06-01 04:17:39.750825] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 8-master-replicate-0: added root inode >[2012-06-01 04:17:39.752460] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 8-master-replicate-1: added root inode >[2012-06-01 04:17:39.757971] I [client.c:2151:notify] 7-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:39.758448] I [client.c:2151:notify] 7-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:39.758495] I [client.c:2151:notify] 7-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:39.758736] I [client.c:2151:notify] 7-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:39.759592] I [client.c:2090:client_rpc_notify] 7-master-client-2: disconnected >[2012-06-01 04:17:39.760200] I [client.c:2090:client_rpc_notify] 7-master-client-0: disconnected >[2012-06-01 04:17:39.760237] I [client.c:2090:client_rpc_notify] 7-master-client-1: disconnected >[2012-06-01 04:17:39.760250] E [afr-common.c:3664:afr_notify] 7-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:39.760357] I [client.c:2090:client_rpc_notify] 7-master-client-3: disconnected >[2012-06-01 04:17:39.760384] E [afr-common.c:3664:afr_notify] 7-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:44.247948] I [io-cache.c:1549:check_cache_size_ok] 9-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:44.248066] I [io-cache.c:1549:check_cache_size_ok] 9-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:44.248159] I [quota.c:3085:quota_parse_limits] 9-master-quota: no "limit-set" option provided >[2012-06-01 04:17:44.255116] I [client.c:2142:notify] 9-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:44.261170] I [client.c:2142:notify] 9-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:44.266096] I [client.c:2142:notify] 9-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:44.270793] I [client.c:2142:notify] 9-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:44.281246] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:44.281390] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:44.281514] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:44.281652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 9-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:17:47.751058] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:47.752491] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:17:47.752525] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:47.752744] I [afr-common.c:3627:afr_notify] 9-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:17:47.755796] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-0: Server lk version = 1 >[2012-06-01 04:17:47.765936] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:47.768692] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:17:47.768736] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:47.769117] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:47.769576] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-1: Server lk version = 1 >[2012-06-01 04:17:47.770085] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:17:47.770113] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:47.770292] I [afr-common.c:3627:afr_notify] 9-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:17:47.770642] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-2: Server lk version = 1 >[2012-06-01 04:17:47.771011] I [client-handshake.c:1636:select_server_supported_programs] 9-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:17:47.772227] I [client-handshake.c:1433:client_setvolume_cbk] 9-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:17:47.772261] I [client-handshake.c:1445:client_setvolume_cbk] 9-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:17:47.777882] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 9 >[2012-06-01 04:17:47.779245] I [client-handshake.c:453:client_set_lk_version_cbk] 9-master-client-3: Server lk version = 1 >[2012-06-01 04:17:47.780941] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 9-master-replicate-0: added root inode >[2012-06-01 04:17:47.782401] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 9-master-replicate-1: added root inode >[2012-06-01 04:17:47.803473] I [client.c:2151:notify] 8-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:47.803536] I [client.c:2151:notify] 8-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:47.803576] I [client.c:2151:notify] 8-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:47.803615] I [client.c:2151:notify] 8-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:17:47.809174] I [client.c:2090:client_rpc_notify] 8-master-client-0: disconnected >[2012-06-01 04:17:47.809226] I [client.c:2090:client_rpc_notify] 8-master-client-1: disconnected >[2012-06-01 04:17:47.809247] E [afr-common.c:3664:afr_notify] 8-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:47.809282] I [client.c:2090:client_rpc_notify] 8-master-client-2: disconnected >[2012-06-01 04:17:47.809318] I [client.c:2090:client_rpc_notify] 8-master-client-3: disconnected >[2012-06-01 04:17:47.809337] E [afr-common.c:3664:afr_notify] 8-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:17:56.410701] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:17:57.447507] I [io-cache.c:1549:check_cache_size_ok] 10-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:17:57.447609] I [io-cache.c:1549:check_cache_size_ok] 10-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:17:57.455377] I [client.c:2142:notify] 10-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:57.459579] I [client.c:2142:notify] 10-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:57.463830] I [client.c:2142:notify] 10-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:17:57.468138] I [client.c:2142:notify] 10-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:17:57.479939] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:17:57.480083] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:17:57.480201] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:17:57.480314] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 10-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:00.779049] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:00.781420] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:00.781453] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:00.781549] I [afr-common.c:3627:afr_notify] 10-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:00.782298] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-0: Server lk version = 1 >[2012-06-01 04:18:00.789068] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:00.792927] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:00.792962] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:00.793847] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-1: Server lk version = 1 >[2012-06-01 04:18:00.795305] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:00.796262] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:00.796295] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:00.796474] I [afr-common.c:3627:afr_notify] 10-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:00.797265] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-2: Server lk version = 1 >[2012-06-01 04:18:00.801531] I [client-handshake.c:1636:select_server_supported_programs] 10-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:00.802979] I [client-handshake.c:1433:client_setvolume_cbk] 10-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:00.803012] I [client-handshake.c:1445:client_setvolume_cbk] 10-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:00.809427] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 10 >[2012-06-01 04:18:00.810150] I [client-handshake.c:453:client_set_lk_version_cbk] 10-master-client-3: Server lk version = 1 >[2012-06-01 04:18:00.813187] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 10-master-replicate-0: added root inode >[2012-06-01 04:18:00.814345] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 10-master-replicate-1: added root inode >[2012-06-01 04:18:00.821239] W [quota.c:2377:quota_fsync_cbk] 9-master-quota: quota context not set in inode (gfid:df187b59-ec7a-46a2-bf61-078ade2e92c9) >[2012-06-01 04:18:00.826075] W [quota.c:2377:quota_fsync_cbk] 9-master-quota: quota context not set in inode (gfid:e5eb198b-c20d-4a32-96a8-ad473e6321a3) >[2012-06-01 04:18:00.829116] I [client.c:2151:notify] 9-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:00.829202] I [client.c:2151:notify] 9-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:00.829249] I [client.c:2151:notify] 9-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:00.829262] I [client.c:2090:client_rpc_notify] 9-master-client-0: disconnected >[2012-06-01 04:18:00.829912] I [client.c:2090:client_rpc_notify] 9-master-client-1: disconnected >[2012-06-01 04:18:00.829944] E [afr-common.c:3664:afr_notify] 9-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:00.829306] I [client.c:2151:notify] 9-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:00.830059] I [client.c:2090:client_rpc_notify] 9-master-client-2: disconnected >[2012-06-01 04:18:00.830600] I [client.c:2090:client_rpc_notify] 9-master-client-3: disconnected >[2012-06-01 04:18:00.830659] E [afr-common.c:3664:afr_notify] 9-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:09.665125] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:10.899068] I [io-cache.c:1549:check_cache_size_ok] 11-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:10.899138] I [io-cache.c:1549:check_cache_size_ok] 11-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:10.899205] I [quota.c:3085:quota_parse_limits] 11-master-quota: no "limit-set" option provided >[2012-06-01 04:18:10.905775] I [client.c:2142:notify] 11-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:10.908908] I [client.c:2142:notify] 11-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:10.912840] I [client.c:2142:notify] 11-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:10.915513] I [client.c:2142:notify] 11-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:10.924832] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:10.924931] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:10.925011] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:10.925101] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 11-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:14.824357] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:14.824637] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:14.826298] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:14.826331] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:14.826530] I [afr-common.c:3627:afr_notify] 11-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:14.826585] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:14.826606] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:14.829726] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-0: Server lk version = 1 >[2012-06-01 04:18:14.829773] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-1: Server lk version = 1 >[2012-06-01 04:18:14.910783] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:14.912618] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:14.912651] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:14.931516] I [afr-common.c:3627:afr_notify] 11-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:14.931676] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-2: Server lk version = 1 >[2012-06-01 04:18:14.959170] I [client-handshake.c:1636:select_server_supported_programs] 11-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:14.961193] I [client-handshake.c:1433:client_setvolume_cbk] 11-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:14.961265] I [client-handshake.c:1445:client_setvolume_cbk] 11-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:15.015400] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 11 >[2012-06-01 04:18:15.017774] I [client-handshake.c:453:client_set_lk_version_cbk] 11-master-client-3: Server lk version = 1 >[2012-06-01 04:18:15.035232] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 11-master-replicate-0: added root inode >[2012-06-01 04:18:15.037483] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 11-master-replicate-1: added root inode >[2012-06-01 04:18:15.055019] I [client.c:2151:notify] 10-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:15.055083] I [client.c:2151:notify] 10-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:15.055114] I [client.c:2151:notify] 10-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:15.055128] I [client.c:2090:client_rpc_notify] 10-master-client-0: disconnected >[2012-06-01 04:18:15.055153] I [client.c:2151:notify] 10-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:15.057972] I [client.c:2090:client_rpc_notify] 10-master-client-1: disconnected >[2012-06-01 04:18:15.057993] E [afr-common.c:3664:afr_notify] 10-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:15.058026] I [client.c:2090:client_rpc_notify] 10-master-client-2: disconnected >[2012-06-01 04:18:15.058051] I [client.c:2090:client_rpc_notify] 10-master-client-3: disconnected >[2012-06-01 04:18:15.058062] E [afr-common.c:3664:afr_notify] 10-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:18.523604] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:19.608540] I [io-cache.c:1549:check_cache_size_ok] 12-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:19.608677] I [io-cache.c:1549:check_cache_size_ok] 12-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:19.615294] I [client.c:2142:notify] 12-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:19.619718] I [client.c:2142:notify] 12-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:19.623898] I [client.c:2142:notify] 12-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:19.628375] I [client.c:2142:notify] 12-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:19.647670] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:19.647898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:19.648019] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:19.648135] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 12-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:22.959586] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:22.962288] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:22.962325] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:22.962523] I [afr-common.c:3627:afr_notify] 12-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:22.964080] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-0: Server lk version = 1 >[2012-06-01 04:18:22.968471] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:22.968998] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:22.969032] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:22.969996] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-1: Server lk version = 1 >[2012-06-01 04:18:22.970820] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:22.972432] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:22.972467] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:22.972567] I [afr-common.c:3627:afr_notify] 12-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:22.972917] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-2: Server lk version = 1 >[2012-06-01 04:18:22.979931] I [client-handshake.c:1636:select_server_supported_programs] 12-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:22.982197] I [client-handshake.c:1433:client_setvolume_cbk] 12-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:22.982234] I [client-handshake.c:1445:client_setvolume_cbk] 12-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:22.987153] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 12 >[2012-06-01 04:18:22.987549] I [client-handshake.c:453:client_set_lk_version_cbk] 12-master-client-3: Server lk version = 1 >[2012-06-01 04:18:22.990499] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 12-master-replicate-0: added root inode >[2012-06-01 04:18:22.993330] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 12-master-replicate-1: added root inode >[2012-06-01 04:18:23.013678] W [quota.c:2377:quota_fsync_cbk] 11-master-quota: quota context not set in inode (gfid:54e7852e-353d-4fa3-9173-40b186a45a4a) >[2012-06-01 04:18:23.022215] I [client.c:2151:notify] 11-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:23.022305] I [client.c:2151:notify] 11-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:23.022357] I [client.c:2151:notify] 11-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:23.022388] I [client.c:2090:client_rpc_notify] 11-master-client-0: disconnected >[2012-06-01 04:18:23.022435] I [client.c:2151:notify] 11-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:23.026174] I [client.c:2090:client_rpc_notify] 11-master-client-1: disconnected >[2012-06-01 04:18:23.026209] E [afr-common.c:3664:afr_notify] 11-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:23.026250] I [client.c:2090:client_rpc_notify] 11-master-client-2: disconnected >[2012-06-01 04:18:23.026290] I [client.c:2090:client_rpc_notify] 11-master-client-3: disconnected >[2012-06-01 04:18:23.026310] E [afr-common.c:3664:afr_notify] 11-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:36.807714] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:37.899306] I [io-cache.c:1549:check_cache_size_ok] 13-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:37.899425] I [io-cache.c:1549:check_cache_size_ok] 13-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:37.899541] I [quota.c:3085:quota_parse_limits] 13-master-quota: no "limit-set" option provided >[2012-06-01 04:18:37.906343] I [client.c:2142:notify] 13-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:37.911385] I [client.c:2142:notify] 13-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:37.915962] I [client.c:2142:notify] 13-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:37.920609] I [client.c:2142:notify] 13-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:37.931293] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:37.931482] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:37.931613] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:37.931742] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 13-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:40.093672] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:40.987060] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:40.992203] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:40.992234] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:40.992407] I [afr-common.c:3627:afr_notify] 13-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:40.992675] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-0: Server lk version = 1 >[2012-06-01 04:18:41.001955] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:41.003827] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:41.003859] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:41.005790] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-1: Server lk version = 1 >[2012-06-01 04:18:41.007010] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:41.009918] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:41.009952] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:41.010125] I [afr-common.c:3627:afr_notify] 13-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:41.012172] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-2: Server lk version = 1 >[2012-06-01 04:18:41.014865] I [client-handshake.c:1636:select_server_supported_programs] 13-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:41.019597] I [client-handshake.c:1433:client_setvolume_cbk] 13-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:41.019655] I [client-handshake.c:1445:client_setvolume_cbk] 13-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:41.024071] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 13 >[2012-06-01 04:18:41.024283] I [client-handshake.c:453:client_set_lk_version_cbk] 13-master-client-3: Server lk version = 1 >[2012-06-01 04:18:41.028211] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 13-master-replicate-0: added root inode >[2012-06-01 04:18:41.030720] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 13-master-replicate-1: added root inode >[2012-06-01 04:18:41.057809] I [client.c:2151:notify] 12-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:41.057916] I [client.c:2151:notify] 12-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:41.057965] I [client.c:2151:notify] 12-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:41.057968] I [client.c:2090:client_rpc_notify] 12-master-client-0: disconnected >[2012-06-01 04:18:41.058028] I [client.c:2151:notify] 12-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:41.062828] I [client.c:2090:client_rpc_notify] 12-master-client-1: disconnected >[2012-06-01 04:18:41.062855] E [afr-common.c:3664:afr_notify] 12-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:41.062924] I [client.c:2090:client_rpc_notify] 12-master-client-2: disconnected >[2012-06-01 04:18:41.062960] I [client.c:2090:client_rpc_notify] 12-master-client-3: disconnected >[2012-06-01 04:18:41.062978] E [afr-common.c:3664:afr_notify] 12-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:41.140520] I [io-cache.c:1549:check_cache_size_ok] 14-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:41.140621] I [io-cache.c:1549:check_cache_size_ok] 14-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:41.149044] I [client.c:2142:notify] 14-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:41.153286] I [client.c:2142:notify] 14-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:41.157258] I [client.c:2142:notify] 14-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:41.161255] I [client.c:2142:notify] 14-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:41.168625] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:41.168757] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:41.168897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:41.169010] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 14-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:43.629894] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:44.809724] I [io-cache.c:1549:check_cache_size_ok] 15-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:44.809802] I [io-cache.c:1549:check_cache_size_ok] 15-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:44.809862] I [quota.c:3085:quota_parse_limits] 15-master-quota: no "limit-set" option provided >[2012-06-01 04:18:44.822705] I [client.c:2142:notify] 15-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:44.830296] I [client.c:2142:notify] 15-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:44.835847] I [client.c:2142:notify] 15-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:44.840980] I [client.c:2142:notify] 15-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:44.849122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:44.849205] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:44.850124] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:44.850200] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 15-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:45.016859] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:45.018335] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:45.018357] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:45.018485] I [afr-common.c:3627:afr_notify] 14-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:45.019886] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-0: Server lk version = 1 >[2012-06-01 04:18:45.027840] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:45.030573] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:45.030595] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:45.031643] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:45.033279] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-1: Server lk version = 1 >[2012-06-01 04:18:45.033461] I [client-handshake.c:1636:select_server_supported_programs] 14-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:45.036592] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:45.036636] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:45.036814] I [afr-common.c:3627:afr_notify] 14-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:45.037213] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-2: Server lk version = 1 >[2012-06-01 04:18:45.037247] I [client-handshake.c:1433:client_setvolume_cbk] 14-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:45.038030] I [client-handshake.c:1445:client_setvolume_cbk] 14-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:45.055261] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 14 >[2012-06-01 04:18:45.056649] I [client-handshake.c:453:client_set_lk_version_cbk] 14-master-client-3: Server lk version = 1 >[2012-06-01 04:18:45.058056] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 14-master-replicate-0: added root inode >[2012-06-01 04:18:45.059204] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 14-master-replicate-1: added root inode >[2012-06-01 04:18:45.084738] W [quota.c:2377:quota_fsync_cbk] 13-master-quota: quota context not set in inode (gfid:d518acaa-c775-41f8-98e3-610ff24709aa) >[2012-06-01 04:18:45.094453] I [client.c:2151:notify] 13-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:45.094519] I [client.c:2151:notify] 13-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:45.094561] I [client.c:2151:notify] 13-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:45.094604] I [client.c:2151:notify] 13-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:45.094557] I [client.c:2090:client_rpc_notify] 13-master-client-0: disconnected >[2012-06-01 04:18:45.097502] I [client.c:2090:client_rpc_notify] 13-master-client-1: disconnected >[2012-06-01 04:18:45.097516] E [afr-common.c:3664:afr_notify] 13-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:45.097547] I [client.c:2090:client_rpc_notify] 13-master-client-2: disconnected >[2012-06-01 04:18:45.097578] I [client.c:2090:client_rpc_notify] 13-master-client-3: disconnected >[2012-06-01 04:18:45.097635] E [afr-common.c:3664:afr_notify] 13-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:48.035951] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:48.037647] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:48.040040] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:48.040132] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:48.041313] I [afr-common.c:3627:afr_notify] 15-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:18:48.041351] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:48.041365] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:48.042990] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-1: Server lk version = 1 >[2012-06-01 04:18:48.043017] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-0: Server lk version = 1 >[2012-06-01 04:18:48.050864] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:48.051722] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:48.051747] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:48.051798] I [afr-common.c:3627:afr_notify] 15-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:48.052464] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-2: Server lk version = 1 >[2012-06-01 04:18:48.067787] I [client-handshake.c:1636:select_server_supported_programs] 15-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:48.069159] I [client-handshake.c:1433:client_setvolume_cbk] 15-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:48.069180] I [client-handshake.c:1445:client_setvolume_cbk] 15-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:48.144970] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 15 >[2012-06-01 04:18:48.145323] I [client-handshake.c:453:client_set_lk_version_cbk] 15-master-client-3: Server lk version = 1 >[2012-06-01 04:18:48.146456] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 15-master-replicate-0: added root inode >[2012-06-01 04:18:48.147069] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 15-master-replicate-1: added root inode >[2012-06-01 04:18:48.165252] I [client.c:2151:notify] 14-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:48.165303] I [client.c:2151:notify] 14-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:48.165332] I [client.c:2151:notify] 14-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:48.165357] I [client.c:2151:notify] 14-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:48.170446] I [client.c:2090:client_rpc_notify] 14-master-client-0: disconnected >[2012-06-01 04:18:48.170496] I [client.c:2090:client_rpc_notify] 14-master-client-1: disconnected >[2012-06-01 04:18:48.170511] E [afr-common.c:3664:afr_notify] 14-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:48.170535] I [client.c:2090:client_rpc_notify] 14-master-client-2: disconnected >[2012-06-01 04:18:48.170558] I [client.c:2090:client_rpc_notify] 14-master-client-3: disconnected >[2012-06-01 04:18:48.170569] E [afr-common.c:3664:afr_notify] 14-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:52.065078] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:53.120492] I [io-cache.c:1549:check_cache_size_ok] 16-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:53.120603] I [io-cache.c:1549:check_cache_size_ok] 16-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:53.127418] I [client.c:2142:notify] 16-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:53.131787] I [client.c:2142:notify] 16-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:53.136282] I [client.c:2142:notify] 16-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:53.140509] I [client.c:2142:notify] 16-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:53.149313] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:53.149408] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:53.149464] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:53.151232] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 16-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:55.809665] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:18:57.003130] I [io-cache.c:1549:check_cache_size_ok] 17-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:18:57.003254] I [io-cache.c:1549:check_cache_size_ok] 17-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:18:57.003343] I [quota.c:3085:quota_parse_limits] 17-master-quota: no "limit-set" option provided >[2012-06-01 04:18:57.026201] I [client.c:2142:notify] 17-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:57.030307] I [client.c:2142:notify] 17-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:57.043456] I [client.c:2142:notify] 17-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:18:57.047614] I [client.c:2142:notify] 17-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:18:57.062607] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:18:57.064991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:18:57.065108] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:18:57.065217] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 17-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:18:57.102416] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:57.108136] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:18:57.108170] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:57.108353] I [afr-common.c:3627:afr_notify] 16-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:18:57.111099] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-0: Server lk version = 1 >[2012-06-01 04:18:57.111163] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:57.115552] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:18:57.115586] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:57.121070] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-1: Server lk version = 1 >[2012-06-01 04:18:57.121334] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:57.127382] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:18:57.127419] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:57.127615] I [afr-common.c:3627:afr_notify] 16-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:18:57.128799] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-2: Server lk version = 1 >[2012-06-01 04:18:57.137942] I [client-handshake.c:1636:select_server_supported_programs] 16-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:18:57.141232] I [client-handshake.c:1433:client_setvolume_cbk] 16-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:18:57.141266] I [client-handshake.c:1445:client_setvolume_cbk] 16-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:18:57.229420] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 16 >[2012-06-01 04:18:57.231190] I [client-handshake.c:453:client_set_lk_version_cbk] 16-master-client-3: Server lk version = 1 >[2012-06-01 04:18:57.234175] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 16-master-replicate-0: added root inode >[2012-06-01 04:18:57.236939] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 16-master-replicate-1: added root inode >[2012-06-01 04:18:57.261543] I [client.c:2151:notify] 15-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:57.261645] I [client.c:2151:notify] 15-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:57.261692] I [client.c:2151:notify] 15-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:57.261737] I [client.c:2151:notify] 15-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:18:57.267768] I [client.c:2090:client_rpc_notify] 15-master-client-0: disconnected >[2012-06-01 04:18:57.267823] I [client.c:2090:client_rpc_notify] 15-master-client-1: disconnected >[2012-06-01 04:18:57.267844] E [afr-common.c:3664:afr_notify] 15-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:57.267912] I [client.c:2090:client_rpc_notify] 15-master-client-2: disconnected >[2012-06-01 04:18:57.267954] I [client.c:2090:client_rpc_notify] 15-master-client-3: disconnected >[2012-06-01 04:18:57.267972] E [afr-common.c:3664:afr_notify] 15-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:18:59.376049] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:00.139388] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:00.141377] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:00.141408] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:00.141602] I [afr-common.c:3627:afr_notify] 17-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:00.142281] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-0: Server lk version = 1 >[2012-06-01 04:19:00.146739] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:00.147422] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:00.147465] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:00.148623] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-1: Server lk version = 1 >[2012-06-01 04:19:00.149747] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:00.150910] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:00.150943] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:00.151122] I [afr-common.c:3627:afr_notify] 17-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:00.152608] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-2: Server lk version = 1 >[2012-06-01 04:19:00.165406] I [client-handshake.c:1636:select_server_supported_programs] 17-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:00.165902] I [client-handshake.c:1433:client_setvolume_cbk] 17-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:00.166783] I [client-handshake.c:1445:client_setvolume_cbk] 17-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:00.172534] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 17 >[2012-06-01 04:19:00.176010] I [client-handshake.c:453:client_set_lk_version_cbk] 17-master-client-3: Server lk version = 1 >[2012-06-01 04:19:00.178085] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 17-master-replicate-0: added root inode >[2012-06-01 04:19:00.180182] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 17-master-replicate-1: added root inode >[2012-06-01 04:19:00.218411] I [client.c:2151:notify] 16-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:00.218502] I [client.c:2151:notify] 16-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:00.218547] I [client.c:2151:notify] 16-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:00.218585] I [client.c:2151:notify] 16-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:00.219675] I [client.c:2090:client_rpc_notify] 16-master-client-0: disconnected >[2012-06-01 04:19:00.219724] I [client.c:2090:client_rpc_notify] 16-master-client-1: disconnected >[2012-06-01 04:19:00.219743] E [afr-common.c:3664:afr_notify] 16-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:00.220486] I [client.c:2090:client_rpc_notify] 16-master-client-2: disconnected >[2012-06-01 04:19:00.221129] I [client.c:2090:client_rpc_notify] 16-master-client-3: disconnected >[2012-06-01 04:19:00.221158] E [afr-common.c:3664:afr_notify] 16-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:00.449145] I [io-cache.c:1549:check_cache_size_ok] 18-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:00.449246] I [io-cache.c:1549:check_cache_size_ok] 18-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:00.456290] I [client.c:2142:notify] 18-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:00.461529] I [client.c:2142:notify] 18-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:00.465596] I [client.c:2142:notify] 18-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:00.470007] I [client.c:2142:notify] 18-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:00.479034] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:00.479161] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:00.479270] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:00.479375] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 18-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:04.166665] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:04.168191] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:04.168217] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:04.168392] I [afr-common.c:3627:afr_notify] 18-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:04.169370] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-0: Server lk version = 1 >[2012-06-01 04:19:04.175776] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:04.176525] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:04.176550] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:04.179206] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-1: Server lk version = 1 >[2012-06-01 04:19:04.179817] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:04.184553] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:04.184580] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:04.184661] I [afr-common.c:3627:afr_notify] 18-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:04.189917] I [client-handshake.c:1636:select_server_supported_programs] 18-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:04.190099] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-2: Server lk version = 1 >[2012-06-01 04:19:04.192043] I [client-handshake.c:1433:client_setvolume_cbk] 18-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:04.192069] I [client-handshake.c:1445:client_setvolume_cbk] 18-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:04.215492] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 18 >[2012-06-01 04:19:04.216604] I [client-handshake.c:453:client_set_lk_version_cbk] 18-master-client-3: Server lk version = 1 >[2012-06-01 04:19:04.219105] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 18-master-replicate-0: added root inode >[2012-06-01 04:19:04.220899] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 18-master-replicate-1: added root inode >[2012-06-01 04:19:04.226696] W [quota.c:2377:quota_fsync_cbk] 17-master-quota: quota context not set in inode (gfid:e12bbfb4-93a7-4301-a3ca-483cd705695b) >[2012-06-01 04:19:04.244852] I [client.c:2151:notify] 17-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:04.244932] I [client.c:2151:notify] 17-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:04.244968] I [client.c:2151:notify] 17-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:04.245000] I [client.c:2151:notify] 17-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:04.247230] I [client.c:2090:client_rpc_notify] 17-master-client-0: disconnected >[2012-06-01 04:19:04.247510] I [client.c:2090:client_rpc_notify] 17-master-client-1: disconnected >[2012-06-01 04:19:04.247533] E [afr-common.c:3664:afr_notify] 17-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:04.247564] I [client.c:2090:client_rpc_notify] 17-master-client-2: disconnected >[2012-06-01 04:19:04.247592] I [client.c:2090:client_rpc_notify] 17-master-client-3: disconnected >[2012-06-01 04:19:04.247606] E [afr-common.c:3664:afr_notify] 17-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:07.670308] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:08.813295] I [io-cache.c:1549:check_cache_size_ok] 19-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:08.813366] I [io-cache.c:1549:check_cache_size_ok] 19-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:08.813417] I [quota.c:3085:quota_parse_limits] 19-master-quota: no "limit-set" option provided >[2012-06-01 04:19:08.826596] I [client.c:2142:notify] 19-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:08.837253] I [client.c:2142:notify] 19-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:08.844252] I [client.c:2142:notify] 19-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:08.864081] I [client.c:2142:notify] 19-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:08.901363] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:08.901857] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:08.903025] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:08.903405] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 19-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:12.199545] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:12.200075] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:12.200469] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:12.200488] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:12.200606] I [afr-common.c:3627:afr_notify] 19-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:19:12.201004] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-1: Server lk version = 1 >[2012-06-01 04:19:12.201034] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:12.201046] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:12.202491] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-0: Server lk version = 1 >[2012-06-01 04:19:12.212521] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:12.212724] I [client-handshake.c:1636:select_server_supported_programs] 19-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:12.212961] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:12.212979] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:12.213087] I [afr-common.c:3627:afr_notify] 19-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:12.213120] I [client-handshake.c:1433:client_setvolume_cbk] 19-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:12.213132] I [client-handshake.c:1445:client_setvolume_cbk] 19-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:12.218925] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 19 >[2012-06-01 04:19:12.219721] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-2: Server lk version = 1 >[2012-06-01 04:19:12.219789] I [client-handshake.c:453:client_set_lk_version_cbk] 19-master-client-3: Server lk version = 1 >[2012-06-01 04:19:12.226089] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 19-master-replicate-0: added root inode >[2012-06-01 04:19:12.229774] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 19-master-replicate-1: added root inode >[2012-06-01 04:19:12.263507] I [client.c:2151:notify] 18-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:12.263572] I [client.c:2151:notify] 18-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:12.263601] I [client.c:2151:notify] 18-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:12.263626] I [client.c:2151:notify] 18-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:12.267390] I [client.c:2090:client_rpc_notify] 18-master-client-0: disconnected >[2012-06-01 04:19:12.267439] I [client.c:2090:client_rpc_notify] 18-master-client-1: disconnected >[2012-06-01 04:19:12.267457] E [afr-common.c:3664:afr_notify] 18-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:12.267492] I [client.c:2090:client_rpc_notify] 18-master-client-2: disconnected >[2012-06-01 04:19:12.267545] I [client.c:2090:client_rpc_notify] 18-master-client-3: disconnected >[2012-06-01 04:19:12.267567] E [afr-common.c:3664:afr_notify] 18-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:16.766570] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:17.798289] I [io-cache.c:1549:check_cache_size_ok] 20-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:17.798369] I [io-cache.c:1549:check_cache_size_ok] 20-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:17.814928] I [client.c:2142:notify] 20-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:17.822027] I [client.c:2142:notify] 20-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:17.827862] I [client.c:2142:notify] 20-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:17.831326] I [client.c:2142:notify] 20-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:17.854671] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:17.855532] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:17.856116] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:17.856199] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 20-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:20.048615] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:21.174145] I [io-cache.c:1549:check_cache_size_ok] 21-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:21.174217] I [io-cache.c:1549:check_cache_size_ok] 21-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:21.174285] I [quota.c:3085:quota_parse_limits] 21-master-quota: no "limit-set" option provided >[2012-06-01 04:19:21.180129] I [client.c:2142:notify] 21-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:21.183032] I [client.c:2142:notify] 21-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:21.185665] I [client.c:2142:notify] 21-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:21.188345] I [client.c:2142:notify] 21-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:21.193685] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:21.193781] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:21.193858] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:21.193956] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 21-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:21.219525] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:21.220802] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:21.220829] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:21.220977] I [afr-common.c:3627:afr_notify] 20-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:21.221645] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-0: Server lk version = 1 >[2012-06-01 04:19:21.230589] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:21.232101] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:21.232123] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:21.232649] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-1: Server lk version = 1 >[2012-06-01 04:19:21.244516] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:21.245118] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:21.245139] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:21.245262] I [afr-common.c:3627:afr_notify] 20-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:21.245795] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-2: Server lk version = 1 >[2012-06-01 04:19:21.255011] I [client-handshake.c:1636:select_server_supported_programs] 20-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:21.255326] I [client-handshake.c:1433:client_setvolume_cbk] 20-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:21.255350] I [client-handshake.c:1445:client_setvolume_cbk] 20-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:21.258982] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 20 >[2012-06-01 04:19:21.260135] I [client-handshake.c:453:client_set_lk_version_cbk] 20-master-client-3: Server lk version = 1 >[2012-06-01 04:19:21.261173] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 20-master-replicate-0: added root inode >[2012-06-01 04:19:21.262308] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 20-master-replicate-1: added root inode >[2012-06-01 04:19:21.268855] W [quota.c:2377:quota_fsync_cbk] 19-master-quota: quota context not set in inode (gfid:fcb017d4-0e48-49a9-a2bc-def38d77c7ea) >[2012-06-01 04:19:21.270969] I [client.c:2151:notify] 19-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:21.271028] I [client.c:2151:notify] 19-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:21.271069] I [client.c:2151:notify] 19-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:21.271096] I [client.c:2151:notify] 19-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:21.276730] I [client.c:2090:client_rpc_notify] 19-master-client-0: disconnected >[2012-06-01 04:19:21.276777] I [client.c:2090:client_rpc_notify] 19-master-client-1: disconnected >[2012-06-01 04:19:21.276790] E [afr-common.c:3664:afr_notify] 19-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:21.276815] I [client.c:2090:client_rpc_notify] 19-master-client-2: disconnected >[2012-06-01 04:19:21.276841] I [client.c:2090:client_rpc_notify] 19-master-client-3: disconnected >[2012-06-01 04:19:21.276852] E [afr-common.c:3664:afr_notify] 19-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:24.255041] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:24.255353] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:24.255374] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:24.255498] I [afr-common.c:3627:afr_notify] 21-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:24.257039] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-0: Server lk version = 1 >[2012-06-01 04:19:24.260899] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:24.261659] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:24.261681] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:24.262430] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-1: Server lk version = 1 >[2012-06-01 04:19:24.264827] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:24.265212] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:24.265234] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:24.265357] I [afr-common.c:3627:afr_notify] 21-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:24.266158] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-2: Server lk version = 1 >[2012-06-01 04:19:24.268690] I [client-handshake.c:1636:select_server_supported_programs] 21-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:24.270194] I [client-handshake.c:1433:client_setvolume_cbk] 21-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:24.270217] I [client-handshake.c:1445:client_setvolume_cbk] 21-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:24.274099] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 21 >[2012-06-01 04:19:24.274694] I [client-handshake.c:453:client_set_lk_version_cbk] 21-master-client-3: Server lk version = 1 >[2012-06-01 04:19:24.276432] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 21-master-replicate-0: added root inode >[2012-06-01 04:19:24.277593] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 21-master-replicate-1: added root inode >[2012-06-01 04:19:24.296487] I [client.c:2151:notify] 20-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:24.296555] I [client.c:2151:notify] 20-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:24.296592] I [client.c:2151:notify] 20-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:24.296644] I [client.c:2151:notify] 20-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:24.299744] I [client.c:2090:client_rpc_notify] 20-master-client-0: disconnected >[2012-06-01 04:19:24.299783] I [client.c:2090:client_rpc_notify] 20-master-client-1: disconnected >[2012-06-01 04:19:24.299798] E [afr-common.c:3664:afr_notify] 20-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:24.299822] I [client.c:2090:client_rpc_notify] 20-master-client-2: disconnected >[2012-06-01 04:19:24.299848] I [client.c:2090:client_rpc_notify] 20-master-client-3: disconnected >[2012-06-01 04:19:24.299860] E [afr-common.c:3664:afr_notify] 20-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:28.335322] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:29.414751] I [io-cache.c:1549:check_cache_size_ok] 22-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:29.414856] I [io-cache.c:1549:check_cache_size_ok] 22-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:29.424898] I [client.c:2142:notify] 22-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:29.434983] I [client.c:2142:notify] 22-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:29.444459] I [client.c:2142:notify] 22-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:29.454947] I [client.c:2142:notify] 22-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:29.473861] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:29.473984] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:29.474327] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:29.474456] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 22-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:33.278064] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:33.279304] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:33.279325] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:33.279532] I [afr-common.c:3627:afr_notify] 22-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:33.280724] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-0: Server lk version = 1 >[2012-06-01 04:19:33.280759] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:33.282178] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:33.282200] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:33.284825] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-1: Server lk version = 1 >[2012-06-01 04:19:33.291118] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:33.292811] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:33.292833] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:33.293074] I [afr-common.c:3627:afr_notify] 22-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:33.293695] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-2: Server lk version = 1 >[2012-06-01 04:19:33.310916] I [client-handshake.c:1636:select_server_supported_programs] 22-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:33.312269] I [client-handshake.c:1433:client_setvolume_cbk] 22-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:33.312291] I [client-handshake.c:1445:client_setvolume_cbk] 22-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:33.344677] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 22 >[2012-06-01 04:19:33.346129] I [client-handshake.c:453:client_set_lk_version_cbk] 22-master-client-3: Server lk version = 1 >[2012-06-01 04:19:33.352895] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 22-master-replicate-0: added root inode >[2012-06-01 04:19:33.355217] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 22-master-replicate-1: added root inode >[2012-06-01 04:19:33.359602] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:56b109e2-1571-42cb-aeb9-d38352a17da0) >[2012-06-01 04:19:33.365200] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:8d1dde71-55d0-4ff8-a328-13e90fd16f92) >[2012-06-01 04:19:33.370070] W [quota.c:2377:quota_fsync_cbk] 21-master-quota: quota context not set in inode (gfid:e9410e81-5eb7-4d96-9f9b-3a0126e5a10a) >[2012-06-01 04:19:33.381895] I [client.c:2151:notify] 21-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:33.381963] I [client.c:2151:notify] 21-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:33.382005] I [client.c:2151:notify] 21-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:33.382035] I [client.c:2151:notify] 21-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:33.385182] I [client.c:2090:client_rpc_notify] 21-master-client-0: disconnected >[2012-06-01 04:19:33.385215] I [client.c:2090:client_rpc_notify] 21-master-client-1: disconnected >[2012-06-01 04:19:33.385227] E [afr-common.c:3664:afr_notify] 21-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:33.385249] I [client.c:2090:client_rpc_notify] 21-master-client-2: disconnected >[2012-06-01 04:19:33.385272] I [client.c:2090:client_rpc_notify] 21-master-client-3: disconnected >[2012-06-01 04:19:33.385282] E [afr-common.c:3664:afr_notify] 21-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:36.553629] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:37.724800] I [io-cache.c:1549:check_cache_size_ok] 23-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:37.724877] I [io-cache.c:1549:check_cache_size_ok] 23-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:37.724935] I [quota.c:3085:quota_parse_limits] 23-master-quota: no "limit-set" option provided >[2012-06-01 04:19:37.740490] I [client.c:2142:notify] 23-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:37.743014] I [client.c:2142:notify] 23-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:37.748333] I [client.c:2142:notify] 23-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:37.755487] I [client.c:2142:notify] 23-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:37.765251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:37.765343] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:37.765546] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:37.765750] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 23-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:40.653436] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:41.323030] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:41.323206] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:41.323838] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:41.324292] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:41.324312] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:41.324427] I [afr-common.c:3627:afr_notify] 23-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:41.324464] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:41.324477] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:41.325107] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-0: Server lk version = 1 >[2012-06-01 04:19:41.325141] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:41.325155] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:41.325208] I [afr-common.c:3627:afr_notify] 23-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:41.325598] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-1: Server lk version = 1 >[2012-06-01 04:19:41.326252] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-2: Server lk version = 1 >[2012-06-01 04:19:41.334418] I [client-handshake.c:1636:select_server_supported_programs] 23-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:41.335826] I [client-handshake.c:1433:client_setvolume_cbk] 23-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:41.335848] I [client-handshake.c:1445:client_setvolume_cbk] 23-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:41.342254] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 23 >[2012-06-01 04:19:41.343730] I [client-handshake.c:453:client_set_lk_version_cbk] 23-master-client-3: Server lk version = 1 >[2012-06-01 04:19:41.346278] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 23-master-replicate-0: added root inode >[2012-06-01 04:19:41.348809] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 23-master-replicate-1: added root inode >[2012-06-01 04:19:41.364752] I [client.c:2151:notify] 22-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:41.364806] I [client.c:2151:notify] 22-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:41.364834] I [client.c:2151:notify] 22-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:41.364874] I [client.c:2151:notify] 22-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:41.368463] I [client.c:2090:client_rpc_notify] 22-master-client-0: disconnected >[2012-06-01 04:19:41.368761] I [client.c:2090:client_rpc_notify] 22-master-client-1: disconnected >[2012-06-01 04:19:41.368777] E [afr-common.c:3664:afr_notify] 22-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:41.368800] I [client.c:2090:client_rpc_notify] 22-master-client-2: disconnected >[2012-06-01 04:19:41.368824] I [client.c:2090:client_rpc_notify] 22-master-client-3: disconnected >[2012-06-01 04:19:41.368835] E [afr-common.c:3664:afr_notify] 22-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:41.735373] I [io-cache.c:1549:check_cache_size_ok] 24-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:41.735450] I [io-cache.c:1549:check_cache_size_ok] 24-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:41.741843] I [client.c:2142:notify] 24-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:41.744387] I [client.c:2142:notify] 24-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:41.746891] I [client.c:2142:notify] 24-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:41.749392] I [client.c:2142:notify] 24-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:41.754704] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:41.754803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:41.754897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:41.754972] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 24-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:44.093061] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:45.346800] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:45.348699] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:45.348732] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:45.352962] I [afr-common.c:3627:afr_notify] 24-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:45.357065] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-0: Server lk version = 1 >[2012-06-01 04:19:45.357111] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:45.359584] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:45.360372] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:45.361939] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-1: Server lk version = 1 >[2012-06-01 04:19:45.361998] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:45.363071] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:45.363104] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:45.363275] I [afr-common.c:3627:afr_notify] 24-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:45.363322] I [client-handshake.c:1636:select_server_supported_programs] 24-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:45.364433] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-2: Server lk version = 1 >[2012-06-01 04:19:45.364496] I [client-handshake.c:1433:client_setvolume_cbk] 24-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:45.364519] I [client-handshake.c:1445:client_setvolume_cbk] 24-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:45.372479] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 24 >[2012-06-01 04:19:45.373704] I [client-handshake.c:453:client_set_lk_version_cbk] 24-master-client-3: Server lk version = 1 >[2012-06-01 04:19:45.375412] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 24-master-replicate-0: added root inode >[2012-06-01 04:19:45.377350] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 24-master-replicate-1: added root inode >[2012-06-01 04:19:45.381595] W [quota.c:2377:quota_fsync_cbk] 23-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b) >[2012-06-01 04:19:45.404453] I [io-cache.c:1549:check_cache_size_ok] 25-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:45.404578] I [io-cache.c:1549:check_cache_size_ok] 25-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:45.404698] I [quota.c:3085:quota_parse_limits] 25-master-quota: no "limit-set" option provided >[2012-06-01 04:19:45.416794] I [client.c:2142:notify] 25-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:45.429920] I [client.c:2142:notify] 25-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:45.434302] I [client.c:2142:notify] 25-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:45.447348] I [client.c:2142:notify] 25-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:45.455002] W [quota.c:2377:quota_fsync_cbk] 23-master-quota: quota context not set in inode (gfid:d5f58101-467e-4b45-b5d4-518a2267eb5c) >[2012-06-01 04:19:45.458045] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:45.459330] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:45.459451] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:45.459764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 25-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:45.478649] I [client.c:2151:notify] 23-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:45.478713] I [client.c:2151:notify] 23-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:45.478755] I [client.c:2151:notify] 23-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:45.478790] I [client.c:2151:notify] 23-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:45.484367] I [client.c:2090:client_rpc_notify] 23-master-client-0: disconnected >[2012-06-01 04:19:45.484518] I [client.c:2090:client_rpc_notify] 23-master-client-1: disconnected >[2012-06-01 04:19:45.484539] E [afr-common.c:3664:afr_notify] 23-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:45.484571] I [client.c:2090:client_rpc_notify] 23-master-client-2: disconnected >[2012-06-01 04:19:45.484603] I [client.c:2090:client_rpc_notify] 23-master-client-3: disconnected >[2012-06-01 04:19:45.484619] E [afr-common.c:3664:afr_notify] 23-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:47.733563] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:48.798395] I [io-cache.c:1549:check_cache_size_ok] 26-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:48.798470] I [io-cache.c:1549:check_cache_size_ok] 26-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:48.808455] I [client.c:2142:notify] 26-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:48.811031] I [client.c:2142:notify] 26-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:48.813580] I [client.c:2142:notify] 26-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:48.816173] I [client.c:2142:notify] 26-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:48.827825] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:48.827897] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:48.827950] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:48.828002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 26-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:49.371668] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:49.380227] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:49.380274] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:49.381280] I [afr-common.c:3627:afr_notify] 25-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:49.382321] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-0: Server lk version = 1 >[2012-06-01 04:19:49.384776] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:49.389656] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:49.389691] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:49.390333] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-1: Server lk version = 1 >[2012-06-01 04:19:49.390696] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:49.397456] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:49.397490] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:49.397726] I [afr-common.c:3627:afr_notify] 25-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:49.398781] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-2: Server lk version = 1 >[2012-06-01 04:19:49.407132] I [client-handshake.c:1636:select_server_supported_programs] 25-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:49.407892] I [client-handshake.c:1433:client_setvolume_cbk] 25-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:49.407924] I [client-handshake.c:1445:client_setvolume_cbk] 25-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:49.635373] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 25 >[2012-06-01 04:19:49.637024] I [client-handshake.c:453:client_set_lk_version_cbk] 25-master-client-3: Server lk version = 1 >[2012-06-01 04:19:49.638278] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 25-master-replicate-0: added root inode >[2012-06-01 04:19:49.639413] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 25-master-replicate-1: added root inode >[2012-06-01 04:19:49.666816] I [client.c:2151:notify] 24-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:49.666879] I [client.c:2151:notify] 24-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:49.666909] I [client.c:2151:notify] 24-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:49.666935] I [client.c:2151:notify] 24-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:49.670369] I [client.c:2090:client_rpc_notify] 24-master-client-0: disconnected >[2012-06-01 04:19:49.670406] I [client.c:2090:client_rpc_notify] 24-master-client-1: disconnected >[2012-06-01 04:19:49.670419] E [afr-common.c:3664:afr_notify] 24-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:49.670445] I [client.c:2090:client_rpc_notify] 24-master-client-2: disconnected >[2012-06-01 04:19:49.670469] I [client.c:2090:client_rpc_notify] 24-master-client-3: disconnected >[2012-06-01 04:19:49.670480] E [afr-common.c:3664:afr_notify] 24-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:51.281431] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:52.408662] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:52.409003] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:52.409030] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:52.409448] I [afr-common.c:3627:afr_notify] 26-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:19:52.409527] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-1: Server lk version = 1 >[2012-06-01 04:19:52.409586] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:52.409970] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:52.410710] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:52.411297] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-0: Server lk version = 1 >[2012-06-01 04:19:52.430716] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:52.431094] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:52.431113] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:52.431227] I [afr-common.c:3627:afr_notify] 26-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:52.432345] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-2: Server lk version = 1 >[2012-06-01 04:19:52.452085] I [client-handshake.c:1636:select_server_supported_programs] 26-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:52.452524] I [client-handshake.c:1433:client_setvolume_cbk] 26-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:52.452544] I [client-handshake.c:1445:client_setvolume_cbk] 26-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:52.469816] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 26 >[2012-06-01 04:19:52.470980] I [client-handshake.c:453:client_set_lk_version_cbk] 26-master-client-3: Server lk version = 1 >[2012-06-01 04:19:52.472581] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 26-master-replicate-0: added root inode >[2012-06-01 04:19:52.474320] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 26-master-replicate-1: added root inode >[2012-06-01 04:19:52.521065] W [quota.c:2377:quota_fsync_cbk] 25-master-quota: quota context not set in inode (gfid:3f282348-54c0-4e62-83c9-a6d9e4bd91f6) >[2012-06-01 04:19:52.526374] W [quota.c:2377:quota_fsync_cbk] 25-master-quota: quota context not set in inode (gfid:978564f8-aa12-48bd-9979-9ddd41eec618) >[2012-06-01 04:19:52.530661] I [client.c:2151:notify] 25-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:52.530713] I [client.c:2151:notify] 25-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:52.530750] I [client.c:2151:notify] 25-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:52.530782] I [client.c:2151:notify] 25-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:52.533639] I [client.c:2090:client_rpc_notify] 25-master-client-0: disconnected >[2012-06-01 04:19:52.533676] I [client.c:2090:client_rpc_notify] 25-master-client-1: disconnected >[2012-06-01 04:19:52.533689] E [afr-common.c:3664:afr_notify] 25-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:52.533718] I [client.c:2090:client_rpc_notify] 25-master-client-2: disconnected >[2012-06-01 04:19:52.533743] I [client.c:2090:client_rpc_notify] 25-master-client-3: disconnected >[2012-06-01 04:19:52.533752] E [afr-common.c:3664:afr_notify] 25-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:52.554749] I [io-cache.c:1549:check_cache_size_ok] 27-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:52.554847] I [io-cache.c:1549:check_cache_size_ok] 27-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:52.562897] I [quota.c:3085:quota_parse_limits] 27-master-quota: no "limit-set" option provided >[2012-06-01 04:19:52.674783] I [client.c:2142:notify] 27-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:52.681630] I [client.c:2142:notify] 27-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:52.707695] I [client.c:2142:notify] 27-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:52.712495] I [client.c:2142:notify] 27-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:52.751606] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:52.751721] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:52.751800] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:52.752306] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 27-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:55.088789] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:56.164014] I [io-cache.c:1549:check_cache_size_ok] 28-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:19:56.164091] I [io-cache.c:1549:check_cache_size_ok] 28-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:19:56.171217] I [client.c:2142:notify] 28-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:56.179404] I [client.c:2142:notify] 28-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:56.184926] I [client.c:2142:notify] 28-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:19:56.191057] I [client.c:2142:notify] 28-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:19:56.198308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:19:56.198429] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:19:56.199066] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:19:56.199148] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 28-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:19:56.457406] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:56.458216] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:56.458241] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:56.458358] I [afr-common.c:3627:afr_notify] 27-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:56.458666] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-0: Server lk version = 1 >[2012-06-01 04:19:56.459267] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:56.459861] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:56.460710] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:56.461034] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-1: Server lk version = 1 >[2012-06-01 04:19:56.471593] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:56.473692] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:56.473713] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:56.473836] I [afr-common.c:3627:afr_notify] 27-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:56.474642] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-2: Server lk version = 1 >[2012-06-01 04:19:56.481300] I [client-handshake.c:1636:select_server_supported_programs] 27-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:56.482943] I [client-handshake.c:1433:client_setvolume_cbk] 27-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:56.482962] I [client-handshake.c:1445:client_setvolume_cbk] 27-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:56.487666] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 27 >[2012-06-01 04:19:56.488692] I [client-handshake.c:453:client_set_lk_version_cbk] 27-master-client-3: Server lk version = 1 >[2012-06-01 04:19:56.489310] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 27-master-replicate-0: added root inode >[2012-06-01 04:19:56.490572] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 27-master-replicate-1: added root inode >[2012-06-01 04:19:56.499652] I [client.c:2151:notify] 26-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:56.499713] I [client.c:2151:notify] 26-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:56.499743] I [client.c:2151:notify] 26-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:56.499778] I [client.c:2151:notify] 26-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:56.502314] I [client.c:2090:client_rpc_notify] 26-master-client-0: disconnected >[2012-06-01 04:19:56.502351] I [client.c:2090:client_rpc_notify] 26-master-client-1: disconnected >[2012-06-01 04:19:56.502363] E [afr-common.c:3664:afr_notify] 26-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:56.502410] I [client.c:2090:client_rpc_notify] 26-master-client-2: disconnected >[2012-06-01 04:19:56.502434] I [client.c:2090:client_rpc_notify] 26-master-client-3: disconnected >[2012-06-01 04:19:56.502445] E [afr-common.c:3664:afr_notify] 26-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:58.491766] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:19:59.490954] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:59.491488] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:59.492527] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:19:59.492550] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:59.492693] I [afr-common.c:3627:afr_notify] 28-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:19:59.493874] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:19:59.493896] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:59.498576] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-0: Server lk version = 1 >[2012-06-01 04:19:59.498608] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-1: Server lk version = 1 >[2012-06-01 04:19:59.502361] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:59.504109] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:19:59.504653] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:59.504780] I [afr-common.c:3627:afr_notify] 28-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:19:59.506241] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-2: Server lk version = 1 >[2012-06-01 04:19:59.511808] I [client-handshake.c:1636:select_server_supported_programs] 28-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:19:59.513912] I [client-handshake.c:1433:client_setvolume_cbk] 28-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:19:59.514546] I [client-handshake.c:1445:client_setvolume_cbk] 28-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:19:59.542017] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 28 >[2012-06-01 04:19:59.544814] I [client-handshake.c:453:client_set_lk_version_cbk] 28-master-client-3: Server lk version = 1 >[2012-06-01 04:19:59.555984] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 28-master-replicate-0: added root inode >[2012-06-01 04:19:59.561016] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 28-master-replicate-1: added root inode >[2012-06-01 04:19:59.584015] W [quota.c:2377:quota_fsync_cbk] 27-master-quota: quota context not set in inode (gfid:8908ad73-a402-4e97-afb5-b551b7fe9359) >[2012-06-01 04:19:59.586125] I [client.c:2151:notify] 27-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:59.586187] I [client.c:2151:notify] 27-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:59.586222] I [client.c:2151:notify] 27-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:59.586251] I [client.c:2151:notify] 27-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:19:59.588353] I [client.c:2090:client_rpc_notify] 27-master-client-0: disconnected >[2012-06-01 04:19:59.588409] I [client.c:2090:client_rpc_notify] 27-master-client-1: disconnected >[2012-06-01 04:19:59.588424] E [afr-common.c:3664:afr_notify] 27-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:19:59.588448] I [client.c:2090:client_rpc_notify] 27-master-client-2: disconnected >[2012-06-01 04:19:59.588471] I [client.c:2090:client_rpc_notify] 27-master-client-3: disconnected >[2012-06-01 04:19:59.588483] E [afr-common.c:3664:afr_notify] 27-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:09.812466] I [io-cache.c:1549:check_cache_size_ok] 29-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:09.812651] I [io-cache.c:1549:check_cache_size_ok] 29-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:09.812714] I [quota.c:3085:quota_parse_limits] 29-master-quota: no "limit-set" option provided >[2012-06-01 04:20:09.819834] I [client.c:2142:notify] 29-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:09.824101] I [client.c:2142:notify] 29-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:09.826971] I [client.c:2142:notify] 29-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:09.829552] I [client.c:2142:notify] 29-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:09.835705] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:09.835812] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:09.835894] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:09.835975] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 29-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:13.003075] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:13.536674] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:13.538057] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:13.538129] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:13.538464] I [afr-common.c:3627:afr_notify] 29-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:13.543082] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-0: Server lk version = 1 >[2012-06-01 04:20:13.545925] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:13.546814] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:13.546844] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:13.548857] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-1: Server lk version = 1 >[2012-06-01 04:20:13.561265] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:13.561912] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:13.562158] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:13.562338] I [afr-common.c:3627:afr_notify] 29-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:13.562768] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-2: Server lk version = 1 >[2012-06-01 04:20:13.567729] I [client-handshake.c:1636:select_server_supported_programs] 29-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:13.569157] I [client-handshake.c:1433:client_setvolume_cbk] 29-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:13.569198] I [client-handshake.c:1445:client_setvolume_cbk] 29-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:13.574083] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 29 >[2012-06-01 04:20:13.574434] I [client-handshake.c:453:client_set_lk_version_cbk] 29-master-client-3: Server lk version = 1 >[2012-06-01 04:20:13.581574] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 29-master-replicate-0: added root inode >[2012-06-01 04:20:13.584979] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 29-master-replicate-1: added root inode >[2012-06-01 04:20:13.634135] I [client.c:2151:notify] 28-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:13.634200] I [client.c:2151:notify] 28-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:13.634261] I [client.c:2151:notify] 28-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:13.634304] I [client.c:2151:notify] 28-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:13.641185] I [client.c:2090:client_rpc_notify] 28-master-client-0: disconnected >[2012-06-01 04:20:13.641238] I [client.c:2090:client_rpc_notify] 28-master-client-1: disconnected >[2012-06-01 04:20:13.641258] E [afr-common.c:3664:afr_notify] 28-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:13.641291] I [client.c:2090:client_rpc_notify] 28-master-client-2: disconnected >[2012-06-01 04:20:13.641325] I [client.c:2090:client_rpc_notify] 28-master-client-3: disconnected >[2012-06-01 04:20:13.641342] E [afr-common.c:3664:afr_notify] 28-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:14.094922] I [io-cache.c:1549:check_cache_size_ok] 30-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:14.095065] I [io-cache.c:1549:check_cache_size_ok] 30-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:14.108310] I [client.c:2142:notify] 30-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:14.121884] I [client.c:2142:notify] 30-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:14.126324] I [client.c:2142:notify] 30-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:14.136263] I [client.c:2142:notify] 30-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:14.159287] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:14.160494] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:14.160708] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:14.160833] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 30-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:16.224023] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:17.396062] I [io-cache.c:1549:check_cache_size_ok] 31-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:17.396142] I [io-cache.c:1549:check_cache_size_ok] 31-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:17.396206] I [quota.c:3085:quota_parse_limits] 31-master-quota: no "limit-set" option provided >[2012-06-01 04:20:17.401562] I [client.c:2142:notify] 31-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:17.416095] I [client.c:2142:notify] 31-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:17.421219] I [client.c:2142:notify] 31-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:17.429778] I [client.c:2142:notify] 31-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:17.444496] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:17.444743] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:17.444818] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:17.444982] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 31-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:17.571642] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:17.572361] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:17.572396] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:17.572661] I [afr-common.c:3627:afr_notify] 30-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:17.575177] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-0: Server lk version = 1 >[2012-06-01 04:20:17.575208] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:17.576218] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:17.576284] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:17.578979] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-1: Server lk version = 1 >[2012-06-01 04:20:17.588895] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:17.589836] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:17.589857] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:17.590417] I [afr-common.c:3627:afr_notify] 30-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:17.596928] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-2: Server lk version = 1 >[2012-06-01 04:20:17.599985] I [client-handshake.c:1636:select_server_supported_programs] 30-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:17.600530] I [client-handshake.c:1433:client_setvolume_cbk] 30-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:17.600551] I [client-handshake.c:1445:client_setvolume_cbk] 30-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:17.608026] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 30 >[2012-06-01 04:20:17.608504] I [client-handshake.c:453:client_set_lk_version_cbk] 30-master-client-3: Server lk version = 1 >[2012-06-01 04:20:17.616461] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 30-master-replicate-0: added root inode >[2012-06-01 04:20:17.618944] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 30-master-replicate-1: added root inode >[2012-06-01 04:20:17.628809] W [quota.c:2377:quota_fsync_cbk] 29-master-quota: quota context not set in inode (gfid:b94ab6ab-6467-42b7-a6d4-e90673e21dbc) >[2012-06-01 04:20:17.652844] I [client.c:2151:notify] 29-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:17.652903] I [client.c:2151:notify] 29-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:17.652961] I [client.c:2151:notify] 29-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:17.653031] I [client.c:2151:notify] 29-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:17.655950] I [socket.c:2315:socket_submit_request] 29-master-client-1: not connected (priv->connected = 255) >[2012-06-01 04:20:17.655970] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-1: failed to submit rpc-request (XID: 0x3889x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-1) >[2012-06-01 04:20:17.655997] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:17.656024] I [socket.c:2315:socket_submit_request] 29-master-client-0: not connected (priv->connected = 255) >[2012-06-01 04:20:17.656065] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-0: failed to submit rpc-request (XID: 0x3885x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-0) >[2012-06-01 04:20:17.656095] W [client3_1-fops.c:881:client3_1_flush_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:17.656166] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-1: failed to submit rpc-request (XID: 0x3890x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-1) >[2012-06-01 04:20:17.656183] W [client3_1-fops.c:881:client3_1_flush_cbk] 29-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:17.656230] W [rpc-clnt.c:1498:rpc_clnt_submit] 29-master-client-0: failed to submit rpc-request (XID: 0x3886x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0) >[2012-06-01 04:20:17.656269] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:17.656315] I [socket.c:1798:socket_event_handler] 29-transport: disconnecting now >[2012-06-01 04:20:17.657770] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 29-master-client-0: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:20:17.651088 (xid=0x3884x) >[2012-06-01 04:20:17.657801] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 29-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:17.657815] I [client.c:2090:client_rpc_notify] 29-master-client-0: disconnected >[2012-06-01 04:20:17.657854] I [socket.c:1798:socket_event_handler] 29-transport: disconnecting now >[2012-06-01 04:20:17.657879] I [client.c:2090:client_rpc_notify] 29-master-client-1: disconnected >[2012-06-01 04:20:17.657891] E [afr-common.c:3664:afr_notify] 29-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:17.658018] I [client.c:2090:client_rpc_notify] 29-master-client-2: disconnected >[2012-06-01 04:20:17.658078] I [client.c:2090:client_rpc_notify] 29-master-client-3: disconnected >[2012-06-01 04:20:17.658114] E [afr-common.c:3664:afr_notify] 29-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:20.617145] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:20.617444] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:20.618261] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:20.618294] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:20.618496] I [afr-common.c:3627:afr_notify] 31-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:20.618552] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:20.620294] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:20.621275] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-0: Server lk version = 1 >[2012-06-01 04:20:20.621319] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-1: Server lk version = 1 >[2012-06-01 04:20:20.642224] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:20.648230] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:20.648265] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:20.648469] I [afr-common.c:3627:afr_notify] 31-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:20.649275] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-2: Server lk version = 1 >[2012-06-01 04:20:20.654315] I [client-handshake.c:1636:select_server_supported_programs] 31-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:20.657040] I [client-handshake.c:1433:client_setvolume_cbk] 31-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:20.657076] I [client-handshake.c:1445:client_setvolume_cbk] 31-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:20.696553] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 31 >[2012-06-01 04:20:20.698301] I [client-handshake.c:453:client_set_lk_version_cbk] 31-master-client-3: Server lk version = 1 >[2012-06-01 04:20:20.704475] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 31-master-replicate-0: added root inode >[2012-06-01 04:20:20.711682] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 31-master-replicate-1: added root inode >[2012-06-01 04:20:20.751317] I [client.c:2151:notify] 30-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:20.751402] I [client.c:2151:notify] 30-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:20.751447] I [client.c:2151:notify] 30-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:20.751485] I [client.c:2151:notify] 30-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:20.757687] I [client.c:2090:client_rpc_notify] 30-master-client-0: disconnected >[2012-06-01 04:20:20.757738] I [client.c:2090:client_rpc_notify] 30-master-client-1: disconnected >[2012-06-01 04:20:20.757758] E [afr-common.c:3664:afr_notify] 30-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:20.757790] I [client.c:2090:client_rpc_notify] 30-master-client-2: disconnected >[2012-06-01 04:20:20.757824] I [client.c:2090:client_rpc_notify] 30-master-client-3: disconnected >[2012-06-01 04:20:20.757841] E [afr-common.c:3664:afr_notify] 30-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:24.791568] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:25.874817] I [io-cache.c:1549:check_cache_size_ok] 32-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:25.874912] I [io-cache.c:1549:check_cache_size_ok] 32-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:25.881543] I [client.c:2142:notify] 32-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:25.884608] I [client.c:2142:notify] 32-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:25.887143] I [client.c:2142:notify] 32-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:25.889552] I [client.c:2142:notify] 32-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:25.895519] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:25.895619] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:25.895722] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:25.895848] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 32-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:28.459730] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:29.660129] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:29.660925] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:29.660946] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:29.661062] I [afr-common.c:3627:afr_notify] 32-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:29.662099] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-0: Server lk version = 1 >[2012-06-01 04:20:29.666368] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:29.667057] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:29.667077] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:29.668714] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-1: Server lk version = 1 >[2012-06-01 04:20:29.677104] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:29.678832] I [client-handshake.c:1636:select_server_supported_programs] 32-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:29.679035] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:29.679055] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:29.679173] I [afr-common.c:3627:afr_notify] 32-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:29.679778] I [client-handshake.c:1433:client_setvolume_cbk] 32-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:29.679798] I [client-handshake.c:1445:client_setvolume_cbk] 32-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:29.687844] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 32 >[2012-06-01 04:20:29.687925] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-2: Server lk version = 1 >[2012-06-01 04:20:29.688796] I [client-handshake.c:453:client_set_lk_version_cbk] 32-master-client-3: Server lk version = 1 >[2012-06-01 04:20:29.693925] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 32-master-replicate-0: added root inode >[2012-06-01 04:20:29.696109] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 32-master-replicate-1: added root inode >[2012-06-01 04:20:29.710238] W [quota.c:2377:quota_fsync_cbk] 31-master-quota: quota context not set in inode (gfid:37ca1a7f-d644-44e9-b0bf-1a62d7380159) >[2012-06-01 04:20:29.712637] I [client.c:2151:notify] 31-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:29.712698] I [client.c:2151:notify] 31-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:29.712729] I [client.c:2151:notify] 31-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:29.712756] I [client.c:2151:notify] 31-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:29.715924] I [client.c:2090:client_rpc_notify] 31-master-client-0: disconnected >[2012-06-01 04:20:29.715962] I [client.c:2090:client_rpc_notify] 31-master-client-1: disconnected >[2012-06-01 04:20:29.715979] E [afr-common.c:3664:afr_notify] 31-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:29.716002] I [client.c:2090:client_rpc_notify] 31-master-client-2: disconnected >[2012-06-01 04:20:29.716024] I [client.c:2090:client_rpc_notify] 31-master-client-3: disconnected >[2012-06-01 04:20:29.716035] E [afr-common.c:3664:afr_notify] 31-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:39.555504] I [io-cache.c:1549:check_cache_size_ok] 33-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:39.555632] I [io-cache.c:1549:check_cache_size_ok] 33-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:39.555745] I [quota.c:3085:quota_parse_limits] 33-master-quota: no "limit-set" option provided >[2012-06-01 04:20:39.563009] I [client.c:2142:notify] 33-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:39.568233] I [client.c:2142:notify] 33-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:39.573438] I [client.c:2142:notify] 33-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:39.579580] I [client.c:2142:notify] 33-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:39.593153] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:39.593308] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:39.593490] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:39.593619] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 33-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:41.924164] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:42.701091] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:42.701359] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:42.702956] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:42.702987] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:42.703177] I [afr-common.c:3627:afr_notify] 33-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:42.703229] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:42.703249] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:42.704569] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-0: Server lk version = 1 >[2012-06-01 04:20:42.704639] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-1: Server lk version = 1 >[2012-06-01 04:20:42.706303] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:42.708765] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:42.708796] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:42.708897] I [afr-common.c:3627:afr_notify] 33-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:42.711522] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-2: Server lk version = 1 >[2012-06-01 04:20:42.717065] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:42.717699] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:42.717729] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:42.723486] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 33 >[2012-06-01 04:20:42.725698] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-3: Server lk version = 1 >[2012-06-01 04:20:42.727237] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 33-master-replicate-0: added root inode >[2012-06-01 04:20:42.728304] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 33-master-replicate-1: added root inode >[2012-06-01 04:20:42.733049] I [fuse-bridge.c:3572:fuse_migrate_fd] 0-glusterfs-fuse: create call on fd (0xf784b14) is in progress, hence waiting >[2012-06-01 04:20:42.963258] I [io-cache.c:1549:check_cache_size_ok] 34-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:42.963361] I [io-cache.c:1549:check_cache_size_ok] 34-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:42.978316] I [client.c:2142:notify] 34-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:42.984907] I [client.c:2142:notify] 34-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:42.999615] I [client.c:2142:notify] 34-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:43.004804] I [client.c:2142:notify] 34-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:43.020345] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:43.020457] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:43.020565] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:43.020660] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 34-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:43.743132] I [client.c:2151:notify] 32-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:43.743227] I [client.c:2151:notify] 32-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:43.743273] I [client.c:2151:notify] 32-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:43.743336] I [client.c:2151:notify] 32-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:43.745709] I [client.c:2090:client_rpc_notify] 32-master-client-0: disconnected >[2012-06-01 04:20:43.745763] I [client.c:2090:client_rpc_notify] 32-master-client-1: disconnected >[2012-06-01 04:20:43.745784] E [afr-common.c:3664:afr_notify] 32-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:43.745819] I [client.c:2090:client_rpc_notify] 32-master-client-2: disconnected >[2012-06-01 04:20:43.745855] I [client.c:2090:client_rpc_notify] 32-master-client-3: disconnected >[2012-06-01 04:20:43.745907] E [afr-common.c:3664:afr_notify] 32-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:45.791230] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:20:46.717734] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:46.719451] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:46.719545] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:46.719750] I [afr-common.c:3627:afr_notify] 34-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:46.720904] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-0: Server lk version = 1 >[2012-06-01 04:20:46.726816] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:46.727936] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:46.727957] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:46.729265] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-1: Server lk version = 1 >[2012-06-01 04:20:46.734516] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:46.736067] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:46.736091] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:46.736367] I [afr-common.c:3627:afr_notify] 34-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:46.736656] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-2: Server lk version = 1 >[2012-06-01 04:20:46.737062] I [client-handshake.c:1636:select_server_supported_programs] 34-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:46.737317] I [client-handshake.c:1433:client_setvolume_cbk] 34-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:46.737395] I [client-handshake.c:1445:client_setvolume_cbk] 34-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:46.742173] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 34 >[2012-06-01 04:20:46.743061] I [client-handshake.c:453:client_set_lk_version_cbk] 34-master-client-3: Server lk version = 1 >[2012-06-01 04:20:46.743887] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 34-master-replicate-0: added root inode >[2012-06-01 04:20:46.745001] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 34-master-replicate-1: added root inode >[2012-06-01 04:20:46.748329] I [client.c:2151:notify] 33-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:46.748591] I [client.c:2151:notify] 33-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:46.748993] I [client.c:2151:notify] 33-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:46.749505] I [client.c:2151:notify] 33-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:46.750656] I [socket.c:1798:socket_event_handler] 33-transport: disconnecting now >[2012-06-01 04:20:46.750735] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 33-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:20:46.748245 (xid=0x1955x) >[2012-06-01 04:20:46.750757] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.750778] I [socket.c:2315:socket_submit_request] 33-master-client-0: not connected (priv->connected = 255) >[2012-06-01 04:20:46.750791] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1957x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-0) >[2012-06-01 04:20:46.750802] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.755106] I [socket.c:2315:socket_submit_request] 33-master-client-1: not connected (priv->connected = 0) >[2012-06-01 04:20:46.755131] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-1: failed to submit rpc-request (XID: 0x1956x Program: GlusterFS 3.1, ProgVers: 330, Proc: 15) to rpc-transport (master-client-1) >[2012-06-01 04:20:46.755144] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.755166] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1958x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0) >[2012-06-01 04:20:46.755186] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.755711] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 33-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FLUSH(15)) called at 2012-06-01 04:20:46.748166 (xid=0x1954x) >[2012-06-01 04:20:46.755732] W [client3_1-fops.c:881:client3_1_flush_cbk] 33-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.755745] I [client.c:2090:client_rpc_notify] 33-master-client-1: disconnected >[2012-06-01 04:20:46.756051] W [rpc-clnt.c:1498:rpc_clnt_submit] 33-master-client-0: failed to submit rpc-request (XID: 0x1959x Program: GlusterFS 3.1, ProgVers: 330, Proc: 30) to rpc-transport (master-client-0) >[2012-06-01 04:20:46.756073] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 33-master-client-0: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:20:46.756087] W [client3_1-fops.c:5267:client3_1_finodelk] 33-master-client-1: (e60426bf-3996-4407-9342-4306d3d71de0) remote_fd is -1. EBADFD >[2012-06-01 04:20:46.756107] I [socket.c:1798:socket_event_handler] 33-transport: disconnecting now >[2012-06-01 04:20:46.756132] I [client.c:2090:client_rpc_notify] 33-master-client-0: disconnected >[2012-06-01 04:20:46.756144] E [afr-common.c:3664:afr_notify] 33-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:46.756316] I [client.c:2090:client_rpc_notify] 33-master-client-2: disconnected >[2012-06-01 04:20:46.756350] I [client.c:2090:client_rpc_notify] 33-master-client-3: disconnected >[2012-06-01 04:20:46.756362] E [afr-common.c:3664:afr_notify] 33-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:46.758797] I [client-handshake.c:1636:select_server_supported_programs] 33-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:46.759643] I [client-handshake.c:1433:client_setvolume_cbk] 33-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:46.759664] I [client-handshake.c:1445:client_setvolume_cbk] 33-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:46.759673] I [client-handshake.c:1282:client_post_handshake] 33-master-client-1: 1 fds open - Delaying child_up until they are re-opened >[2012-06-01 04:20:46.762023] I [client-lk.c:601:decrement_reopen_fd_count] 33-master-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP >[2012-06-01 04:20:46.762110] I [afr-common.c:3627:afr_notify] 33-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:20:46.764159] I [client-handshake.c:453:client_set_lk_version_cbk] 33-master-client-1: Server lk version = 1 >[2012-06-01 04:20:46.953794] I [io-cache.c:1549:check_cache_size_ok] 35-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:20:46.953872] I [io-cache.c:1549:check_cache_size_ok] 35-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:20:46.953944] I [quota.c:3085:quota_parse_limits] 35-master-quota: no "limit-set" option provided >[2012-06-01 04:20:46.961880] I [client.c:2142:notify] 35-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:46.966151] I [client.c:2142:notify] 35-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:46.968863] I [client.c:2142:notify] 35-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:20:46.971438] I [client.c:2142:notify] 35-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:20:46.981157] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:20:46.981254] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:20:46.981337] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:20:46.981411] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 35-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:20:50.747530] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:50.747944] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:20:50.747967] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:50.748086] I [afr-common.c:3627:afr_notify] 35-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:20:50.750859] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-0: Server lk version = 1 >[2012-06-01 04:20:50.755708] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:50.756118] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:20:50.756139] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:50.759015] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-1: Server lk version = 1 >[2012-06-01 04:20:50.760056] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:50.762671] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:20:50.762692] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:50.763497] I [afr-common.c:3627:afr_notify] 35-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:20:50.765084] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-2: Server lk version = 1 >[2012-06-01 04:20:50.768128] I [client-handshake.c:1636:select_server_supported_programs] 35-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:20:50.768709] I [client-handshake.c:1433:client_setvolume_cbk] 35-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:20:50.768730] I [client-handshake.c:1445:client_setvolume_cbk] 35-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:20:50.798994] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 35 >[2012-06-01 04:20:50.799481] I [client-handshake.c:453:client_set_lk_version_cbk] 35-master-client-3: Server lk version = 1 >[2012-06-01 04:20:50.800424] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 35-master-replicate-0: added root inode >[2012-06-01 04:20:50.803270] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 35-master-replicate-1: added root inode >[2012-06-01 04:20:50.816878] I [client.c:2151:notify] 34-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:50.816927] I [client.c:2151:notify] 34-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:50.816954] I [client.c:2151:notify] 34-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:50.816982] I [client.c:2151:notify] 34-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:20:50.821042] I [client.c:2090:client_rpc_notify] 34-master-client-0: disconnected >[2012-06-01 04:20:50.821080] I [client.c:2090:client_rpc_notify] 34-master-client-1: disconnected >[2012-06-01 04:20:50.821093] E [afr-common.c:3664:afr_notify] 34-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:50.821118] I [client.c:2090:client_rpc_notify] 34-master-client-2: disconnected >[2012-06-01 04:20:50.821143] I [client.c:2090:client_rpc_notify] 34-master-client-3: disconnected >[2012-06-01 04:20:50.821156] E [afr-common.c:3664:afr_notify] 34-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:20:59.493594] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:00.580731] I [io-cache.c:1549:check_cache_size_ok] 36-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:00.580808] I [io-cache.c:1549:check_cache_size_ok] 36-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:00.642068] I [client.c:2142:notify] 36-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:00.651639] I [client.c:2142:notify] 36-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:00.654683] I [client.c:2142:notify] 36-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:00.665665] I [client.c:2142:notify] 36-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:00.675506] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:00.676165] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:00.676251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:00.676402] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 36-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:03.107194] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:03.783749] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:03.783827] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:03.784558] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:03.785390] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:03.785410] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:03.785521] I [afr-common.c:3627:afr_notify] 36-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:03.785564] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:03.785578] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:03.785958] I [afr-common.c:3627:afr_notify] 36-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:03.785995] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:03.786008] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:03.786984] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-2: Server lk version = 1 >[2012-06-01 04:21:03.787018] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-0: Server lk version = 1 >[2012-06-01 04:21:03.787047] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-1: Server lk version = 1 >[2012-06-01 04:21:03.795608] I [client-handshake.c:1636:select_server_supported_programs] 36-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:03.795896] I [client-handshake.c:1433:client_setvolume_cbk] 36-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:03.795915] I [client-handshake.c:1445:client_setvolume_cbk] 36-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:03.801552] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 36 >[2012-06-01 04:21:03.802327] I [client-handshake.c:453:client_set_lk_version_cbk] 36-master-client-3: Server lk version = 1 >[2012-06-01 04:21:03.804721] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 36-master-replicate-0: added root inode >[2012-06-01 04:21:03.806026] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 36-master-replicate-1: added root inode >[2012-06-01 04:21:03.824086] I [client.c:2151:notify] 35-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:03.824146] I [client.c:2151:notify] 35-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:03.824174] I [client.c:2151:notify] 35-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:03.824197] I [client.c:2151:notify] 35-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:03.827130] I [client.c:2090:client_rpc_notify] 35-master-client-0: disconnected >[2012-06-01 04:21:03.827165] I [client.c:2090:client_rpc_notify] 35-master-client-1: disconnected >[2012-06-01 04:21:03.827177] E [afr-common.c:3664:afr_notify] 35-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:03.827413] I [client.c:2090:client_rpc_notify] 35-master-client-2: disconnected >[2012-06-01 04:21:03.827442] I [client.c:2090:client_rpc_notify] 35-master-client-3: disconnected >[2012-06-01 04:21:03.827453] E [afr-common.c:3664:afr_notify] 35-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:04.220817] I [io-cache.c:1549:check_cache_size_ok] 37-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:04.220897] I [io-cache.c:1549:check_cache_size_ok] 37-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:04.220960] I [quota.c:3085:quota_parse_limits] 37-master-quota: no "limit-set" option provided >[2012-06-01 04:21:04.318148] I [client.c:2142:notify] 37-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:04.326512] I [client.c:2142:notify] 37-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:04.329949] I [client.c:2142:notify] 37-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:04.333644] I [client.c:2142:notify] 37-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:04.340085] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:04.340179] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:04.340257] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:04.340333] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 37-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:06.667242] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:07.747770] I [io-cache.c:1549:check_cache_size_ok] 38-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:07.747842] I [io-cache.c:1549:check_cache_size_ok] 38-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:07.759300] I [client.c:2142:notify] 38-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:07.768099] I [client.c:2142:notify] 38-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:07.770763] I [client.c:2142:notify] 38-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:07.773318] I [client.c:2142:notify] 38-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:07.787644] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:07.787731] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:07.788225] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:07.788310] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 38-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:07.798748] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:07.800684] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:07.800817] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:07.801379] I [afr-common.c:3627:afr_notify] 37-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:07.801447] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-0: Server lk version = 1 >[2012-06-01 04:21:07.809572] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:07.810053] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:07.810072] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:07.813893] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-1: Server lk version = 1 >[2012-06-01 04:21:07.813922] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:07.814299] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:07.814317] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:07.814421] I [afr-common.c:3627:afr_notify] 37-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:07.815254] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-2: Server lk version = 1 >[2012-06-01 04:21:07.815922] I [client-handshake.c:1636:select_server_supported_programs] 37-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:07.817065] I [client-handshake.c:1433:client_setvolume_cbk] 37-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:07.817085] I [client-handshake.c:1445:client_setvolume_cbk] 37-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:07.835233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 37 >[2012-06-01 04:21:07.837045] I [client-handshake.c:453:client_set_lk_version_cbk] 37-master-client-3: Server lk version = 1 >[2012-06-01 04:21:07.840303] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 37-master-replicate-0: added root inode >[2012-06-01 04:21:07.843308] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 37-master-replicate-1: added root inode >[2012-06-01 04:21:07.868585] I [client.c:2151:notify] 36-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:07.868670] I [client.c:2151:notify] 36-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:07.868700] I [client.c:2151:notify] 36-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:07.868735] I [client.c:2151:notify] 36-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:07.872537] I [client.c:2090:client_rpc_notify] 36-master-client-0: disconnected >[2012-06-01 04:21:07.872573] I [client.c:2090:client_rpc_notify] 36-master-client-1: disconnected >[2012-06-01 04:21:07.872586] E [afr-common.c:3664:afr_notify] 36-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:07.872638] I [client.c:2090:client_rpc_notify] 36-master-client-2: disconnected >[2012-06-01 04:21:07.872665] I [client.c:2090:client_rpc_notify] 36-master-client-3: disconnected >[2012-06-01 04:21:07.872676] E [afr-common.c:3664:afr_notify] 36-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:10.050508] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:10.820528] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:10.823160] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:10.823185] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:10.823228] I [afr-common.c:3627:afr_notify] 38-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:10.826341] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-0: Server lk version = 1 >[2012-06-01 04:21:10.826376] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:10.827248] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:10.827269] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:10.830696] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-1: Server lk version = 1 >[2012-06-01 04:21:10.832432] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:10.832662] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:10.833112] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:10.833243] I [afr-common.c:3627:afr_notify] 38-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:21:10.834881] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-3: Server lk version = 1 >[2012-06-01 04:21:10.835704] I [client-handshake.c:1636:select_server_supported_programs] 38-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:10.836788] I [client-handshake.c:1433:client_setvolume_cbk] 38-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:10.836826] I [client-handshake.c:1445:client_setvolume_cbk] 38-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:10.841057] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 38 >[2012-06-01 04:21:10.841855] I [client-handshake.c:453:client_set_lk_version_cbk] 38-master-client-2: Server lk version = 1 >[2012-06-01 04:21:10.844659] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 38-master-replicate-0: added root inode >[2012-06-01 04:21:10.846483] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 38-master-replicate-1: added root inode >[2012-06-01 04:21:10.865280] W [quota.c:2377:quota_fsync_cbk] 37-master-quota: quota context not set in inode (gfid:c47d718d-1349-420e-83a3-2d6a59c22f7b) >[2012-06-01 04:21:10.867579] I [client.c:2151:notify] 37-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:10.867647] I [client.c:2151:notify] 37-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:10.867686] I [client.c:2151:notify] 37-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:10.867713] I [client.c:2151:notify] 37-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:10.873703] I [client.c:2090:client_rpc_notify] 37-master-client-0: disconnected >[2012-06-01 04:21:10.873752] I [client.c:2090:client_rpc_notify] 37-master-client-1: disconnected >[2012-06-01 04:21:10.873766] E [afr-common.c:3664:afr_notify] 37-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:10.873790] I [client.c:2090:client_rpc_notify] 37-master-client-2: disconnected >[2012-06-01 04:21:10.873812] I [client.c:2090:client_rpc_notify] 37-master-client-3: disconnected >[2012-06-01 04:21:10.873823] E [afr-common.c:3664:afr_notify] 37-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:11.167142] I [io-cache.c:1549:check_cache_size_ok] 39-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:11.167241] I [io-cache.c:1549:check_cache_size_ok] 39-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:11.167302] I [quota.c:3085:quota_parse_limits] 39-master-quota: no "limit-set" option provided >[2012-06-01 04:21:11.174682] I [client.c:2142:notify] 39-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:11.179096] I [client.c:2142:notify] 39-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:11.188381] I [client.c:2142:notify] 39-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:11.192787] I [client.c:2142:notify] 39-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:11.203853] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:11.203996] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:11.204107] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:11.204214] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 39-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:13.515183] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:14.566351] I [io-cache.c:1549:check_cache_size_ok] 40-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:14.566438] I [io-cache.c:1549:check_cache_size_ok] 40-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:14.696669] I [client.c:2142:notify] 40-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:14.707709] I [client.c:2142:notify] 40-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:14.721787] I [client.c:2142:notify] 40-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:14.726648] I [client.c:2142:notify] 40-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:14.740256] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:14.740563] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:14.740666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:14.740954] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 40-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:14.835874] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:14.836967] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:14.836987] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:14.837101] I [afr-common.c:3627:afr_notify] 39-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:14.837710] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-0: Server lk version = 1 >[2012-06-01 04:21:14.840149] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:14.841603] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:14.841643] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:14.842883] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-1: Server lk version = 1 >[2012-06-01 04:21:14.852486] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:14.855474] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:14.855495] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:14.855608] I [afr-common.c:3627:afr_notify] 39-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:14.857465] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-2: Server lk version = 1 >[2012-06-01 04:21:14.858490] I [client-handshake.c:1636:select_server_supported_programs] 39-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:14.859345] I [client-handshake.c:1433:client_setvolume_cbk] 39-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:14.859365] I [client-handshake.c:1445:client_setvolume_cbk] 39-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:14.868687] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 39 >[2012-06-01 04:21:14.869814] I [client-handshake.c:453:client_set_lk_version_cbk] 39-master-client-3: Server lk version = 1 >[2012-06-01 04:21:14.870325] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 39-master-replicate-0: added root inode >[2012-06-01 04:21:14.873223] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 39-master-replicate-1: added root inode >[2012-06-01 04:21:14.900131] I [client.c:2151:notify] 38-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:14.900183] I [client.c:2151:notify] 38-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:14.900210] I [client.c:2151:notify] 38-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:14.900234] I [client.c:2151:notify] 38-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:14.905044] I [client.c:2090:client_rpc_notify] 38-master-client-0: disconnected >[2012-06-01 04:21:14.905081] I [client.c:2090:client_rpc_notify] 38-master-client-1: disconnected >[2012-06-01 04:21:14.905094] E [afr-common.c:3664:afr_notify] 38-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:14.905115] I [client.c:2090:client_rpc_notify] 38-master-client-2: disconnected >[2012-06-01 04:21:14.905138] I [client.c:2090:client_rpc_notify] 38-master-client-3: disconnected >[2012-06-01 04:21:14.905148] E [afr-common.c:3664:afr_notify] 38-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:17.140304] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:17.867835] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:17.868411] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:17.869977] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:17.869998] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:17.870115] I [afr-common.c:3627:afr_notify] 40-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:17.871598] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-0: Server lk version = 1 >[2012-06-01 04:21:17.871633] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:17.871647] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:17.872166] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-1: Server lk version = 1 >[2012-06-01 04:21:17.881267] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:17.882077] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:17.882100] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:17.882700] I [afr-common.c:3627:afr_notify] 40-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:17.885268] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-2: Server lk version = 1 >[2012-06-01 04:21:17.893307] I [client-handshake.c:1636:select_server_supported_programs] 40-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:17.894461] I [client-handshake.c:1433:client_setvolume_cbk] 40-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:17.894492] I [client-handshake.c:1445:client_setvolume_cbk] 40-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:17.923460] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 40 >[2012-06-01 04:21:17.928309] I [client-handshake.c:453:client_set_lk_version_cbk] 40-master-client-3: Server lk version = 1 >[2012-06-01 04:21:17.931119] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 40-master-replicate-0: added root inode >[2012-06-01 04:21:17.932271] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 40-master-replicate-1: added root inode >[2012-06-01 04:21:17.951788] W [quota.c:2377:quota_fsync_cbk] 39-master-quota: quota context not set in inode (gfid:8056796e-06f9-4de2-8d4e-9cd1e1e22b21) >[2012-06-01 04:21:17.958549] I [client.c:2151:notify] 39-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:17.958609] I [client.c:2151:notify] 39-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:17.958657] I [client.c:2151:notify] 39-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:17.958681] I [client.c:2151:notify] 39-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:17.961774] I [client.c:2090:client_rpc_notify] 39-master-client-0: disconnected >[2012-06-01 04:21:17.961810] I [client.c:2090:client_rpc_notify] 39-master-client-1: disconnected >[2012-06-01 04:21:17.961823] E [afr-common.c:3664:afr_notify] 39-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:17.961845] I [client.c:2090:client_rpc_notify] 39-master-client-2: disconnected >[2012-06-01 04:21:17.961867] I [client.c:2090:client_rpc_notify] 39-master-client-3: disconnected >[2012-06-01 04:21:17.961878] E [afr-common.c:3664:afr_notify] 39-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:18.375171] I [io-cache.c:1549:check_cache_size_ok] 41-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:18.375299] I [io-cache.c:1549:check_cache_size_ok] 41-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:18.375362] I [quota.c:3085:quota_parse_limits] 41-master-quota: no "limit-set" option provided >[2012-06-01 04:21:18.407008] I [client.c:2142:notify] 41-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:18.409447] I [client.c:2142:notify] 41-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:18.418114] I [client.c:2142:notify] 41-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:18.428746] I [client.c:2142:notify] 41-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:18.454419] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:18.454511] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:18.454585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:18.454678] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 41-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:20.666556] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:21.766005] I [io-cache.c:1549:check_cache_size_ok] 42-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:21.766082] I [io-cache.c:1549:check_cache_size_ok] 42-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:21.792719] I [client.c:2142:notify] 42-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:21.804276] I [client.c:2142:notify] 42-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:21.815859] I [client.c:2142:notify] 42-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:21.829650] I [client.c:2142:notify] 42-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:21.845992] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:21.846656] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:21.846978] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:21.847409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 42-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:21.905424] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:21.915351] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:21.915381] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:21.916014] I [afr-common.c:3627:afr_notify] 41-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:21.916123] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-0: Server lk version = 1 >[2012-06-01 04:21:21.921296] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:21.924777] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:21.924799] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:21.927125] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-1: Server lk version = 1 >[2012-06-01 04:21:21.954798] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:21.955794] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:21.955815] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:21.955970] I [afr-common.c:3627:afr_notify] 41-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:21.957800] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-2: Server lk version = 1 >[2012-06-01 04:21:21.998397] I [client-handshake.c:1636:select_server_supported_programs] 41-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:21.999570] I [client-handshake.c:1433:client_setvolume_cbk] 41-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:21.999595] I [client-handshake.c:1445:client_setvolume_cbk] 41-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:22.106387] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 41 >[2012-06-01 04:21:22.107219] I [client-handshake.c:453:client_set_lk_version_cbk] 41-master-client-3: Server lk version = 1 >[2012-06-01 04:21:22.113256] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 41-master-replicate-0: added root inode >[2012-06-01 04:21:22.114079] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 41-master-replicate-1: added root inode >[2012-06-01 04:21:22.199203] I [client.c:2151:notify] 40-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:22.199263] I [client.c:2151:notify] 40-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:22.199292] I [client.c:2151:notify] 40-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:22.199318] I [client.c:2151:notify] 40-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:22.205047] I [client.c:2090:client_rpc_notify] 40-master-client-0: disconnected >[2012-06-01 04:21:22.205089] I [client.c:2090:client_rpc_notify] 40-master-client-1: disconnected >[2012-06-01 04:21:22.205102] E [afr-common.c:3664:afr_notify] 40-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:22.205126] I [client.c:2090:client_rpc_notify] 40-master-client-2: disconnected >[2012-06-01 04:21:22.205150] I [client.c:2090:client_rpc_notify] 40-master-client-3: disconnected >[2012-06-01 04:21:22.205161] E [afr-common.c:3664:afr_notify] 40-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:24.404385] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:24.987761] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:24.988650] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:24.988671] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:24.988785] I [afr-common.c:3627:afr_notify] 42-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:24.991442] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-0: Server lk version = 1 >[2012-06-01 04:21:24.997476] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:25.003386] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:25.003410] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:25.003992] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:25.004675] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-1: Server lk version = 1 >[2012-06-01 04:21:25.004713] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:25.004769] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:25.004888] I [afr-common.c:3627:afr_notify] 42-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:25.010445] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-2: Server lk version = 1 >[2012-06-01 04:21:25.013538] I [client-handshake.c:1636:select_server_supported_programs] 42-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:25.014198] I [client-handshake.c:1433:client_setvolume_cbk] 42-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:25.014219] I [client-handshake.c:1445:client_setvolume_cbk] 42-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:25.030828] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 42 >[2012-06-01 04:21:25.031347] I [client-handshake.c:453:client_set_lk_version_cbk] 42-master-client-3: Server lk version = 1 >[2012-06-01 04:21:25.036076] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 42-master-replicate-0: added root inode >[2012-06-01 04:21:25.037504] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 42-master-replicate-1: added root inode >[2012-06-01 04:21:25.042519] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:665c9158-d394-4999-88fc-f134a1f4422e) >[2012-06-01 04:21:25.047014] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:d58a6538-1af4-41eb-ae3b-0e6ef60dd97c) >[2012-06-01 04:21:25.051878] W [quota.c:2377:quota_fsync_cbk] 41-master-quota: quota context not set in inode (gfid:9cd8b3d8-7e7a-45c0-8b3d-a2600e946953) >[2012-06-01 04:21:25.055540] I [client.c:2151:notify] 41-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:25.055589] I [client.c:2151:notify] 41-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:25.055633] I [client.c:2151:notify] 41-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:25.055662] I [client.c:2151:notify] 41-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:25.059690] I [client.c:2090:client_rpc_notify] 41-master-client-0: disconnected >[2012-06-01 04:21:25.059733] I [client.c:2090:client_rpc_notify] 41-master-client-1: disconnected >[2012-06-01 04:21:25.059747] E [afr-common.c:3664:afr_notify] 41-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:25.059771] I [client.c:2090:client_rpc_notify] 41-master-client-2: disconnected >[2012-06-01 04:21:25.059798] I [client.c:2090:client_rpc_notify] 41-master-client-3: disconnected >[2012-06-01 04:21:25.059809] E [afr-common.c:3664:afr_notify] 41-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:25.610050] I [io-cache.c:1549:check_cache_size_ok] 43-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:25.610122] I [io-cache.c:1549:check_cache_size_ok] 43-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:25.610190] I [quota.c:3085:quota_parse_limits] 43-master-quota: no "limit-set" option provided >[2012-06-01 04:21:25.636506] I [client.c:2142:notify] 43-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:25.642832] I [client.c:2142:notify] 43-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:25.650635] I [client.c:2142:notify] 43-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:25.655432] I [client.c:2142:notify] 43-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:25.665336] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:25.665418] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:25.665492] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:25.667277] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 43-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:28.101498] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:29.016654] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:29.026402] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:29.026424] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:29.028079] I [afr-common.c:3627:afr_notify] 43-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:29.030045] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-0: Server lk version = 1 >[2012-06-01 04:21:29.031087] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:29.033092] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:29.033112] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:29.033953] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:29.034840] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-1: Server lk version = 1 >[2012-06-01 04:21:29.034892] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:29.034911] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:29.035140] I [afr-common.c:3627:afr_notify] 43-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:29.036482] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-2: Server lk version = 1 >[2012-06-01 04:21:29.051775] I [client-handshake.c:1636:select_server_supported_programs] 43-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:29.052718] I [client-handshake.c:1433:client_setvolume_cbk] 43-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:29.052741] I [client-handshake.c:1445:client_setvolume_cbk] 43-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:29.092151] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 43 >[2012-06-01 04:21:29.093676] I [client-handshake.c:453:client_set_lk_version_cbk] 43-master-client-3: Server lk version = 1 >[2012-06-01 04:21:29.096785] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 43-master-replicate-0: added root inode >[2012-06-01 04:21:29.098080] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 43-master-replicate-1: added root inode >[2012-06-01 04:21:29.110498] I [client.c:2151:notify] 42-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:29.110681] I [client.c:2151:notify] 42-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:29.110850] I [client.c:2151:notify] 42-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:29.110977] I [client.c:2151:notify] 42-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:29.112011] I [socket.c:1798:socket_event_handler] 42-transport: disconnecting now >[2012-06-01 04:21:29.112083] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x3b4180f547] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) [0x3b4180ef42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3b4180ebfe]))) 42-master-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-06-01 04:21:29.107297 (xid=0x2530x) >[2012-06-01 04:21:29.112105] W [client3_1-fops.c:1545:client3_1_finodelk_cbk] 42-master-client-1: remote operation failed: Transport endpoint is not connected >[2012-06-01 04:21:29.112133] I [client.c:2090:client_rpc_notify] 42-master-client-1: disconnected >[2012-06-01 04:21:29.112162] I [client.c:2090:client_rpc_notify] 42-master-client-2: disconnected >[2012-06-01 04:21:29.112184] I [client.c:2090:client_rpc_notify] 42-master-client-0: disconnected >[2012-06-01 04:21:29.112204] E [afr-common.c:3664:afr_notify] 42-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:29.112473] I [client.c:2090:client_rpc_notify] 42-master-client-3: disconnected >[2012-06-01 04:21:29.112492] E [afr-common.c:3664:afr_notify] 42-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:34.261159] I [io-cache.c:1549:check_cache_size_ok] 44-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:34.261240] I [io-cache.c:1549:check_cache_size_ok] 44-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:34.345146] I [client.c:2142:notify] 44-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:34.350750] I [client.c:2142:notify] 44-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:34.356755] I [client.c:2142:notify] 44-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:34.362546] I [client.c:2142:notify] 44-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:34.392398] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:34.394364] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:34.394452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:34.396954] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 44-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:36.738499] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:37.913315] I [io-cache.c:1549:check_cache_size_ok] 45-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:37.913387] I [io-cache.c:1549:check_cache_size_ok] 45-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:37.913443] I [quota.c:3085:quota_parse_limits] 45-master-quota: no "limit-set" option provided >[2012-06-01 04:21:37.924646] I [client.c:2142:notify] 45-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:37.932405] I [client.c:2142:notify] 45-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:37.937839] I [client.c:2142:notify] 45-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:37.945231] I [client.c:2142:notify] 45-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:37.956134] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:37.956372] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:37.956466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:37.956779] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 45-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:38.058756] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:38.059803] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:38.059825] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:38.060342] I [afr-common.c:3627:afr_notify] 44-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:38.061851] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-0: Server lk version = 1 >[2012-06-01 04:21:38.066559] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:38.070469] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:38.070492] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:38.080383] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-1: Server lk version = 1 >[2012-06-01 04:21:38.081807] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:38.085458] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:38.085481] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:38.085525] I [afr-common.c:3627:afr_notify] 44-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:38.089110] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-2: Server lk version = 1 >[2012-06-01 04:21:38.130237] I [client-handshake.c:1636:select_server_supported_programs] 44-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:38.132662] I [client-handshake.c:1433:client_setvolume_cbk] 44-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:38.132689] I [client-handshake.c:1445:client_setvolume_cbk] 44-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:38.146217] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 44 >[2012-06-01 04:21:38.146386] I [client-handshake.c:453:client_set_lk_version_cbk] 44-master-client-3: Server lk version = 1 >[2012-06-01 04:21:38.151578] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 44-master-replicate-0: added root inode >[2012-06-01 04:21:38.152447] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 44-master-replicate-1: added root inode >[2012-06-01 04:21:38.165401] I [client.c:2151:notify] 43-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:38.165457] I [client.c:2151:notify] 43-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:38.165489] I [client.c:2151:notify] 43-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:38.165526] I [client.c:2151:notify] 43-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:38.169125] I [client.c:2090:client_rpc_notify] 43-master-client-0: disconnected >[2012-06-01 04:21:38.169163] I [client.c:2090:client_rpc_notify] 43-master-client-1: disconnected >[2012-06-01 04:21:38.169176] E [afr-common.c:3664:afr_notify] 43-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:38.169199] I [client.c:2090:client_rpc_notify] 43-master-client-2: disconnected >[2012-06-01 04:21:38.169222] I [client.c:2090:client_rpc_notify] 43-master-client-3: disconnected >[2012-06-01 04:21:38.169234] E [afr-common.c:3664:afr_notify] 43-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:41.121839] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:41.124034] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:41.124054] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:41.124171] I [afr-common.c:3627:afr_notify] 45-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:21:41.126296] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-0: Server lk version = 1 >[2012-06-01 04:21:41.126344] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:41.126584] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:41.126604] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:41.132366] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-1: Server lk version = 1 >[2012-06-01 04:21:41.132399] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:41.134858] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:41.134881] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:41.135002] I [afr-common.c:3627:afr_notify] 45-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:41.135646] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-2: Server lk version = 1 >[2012-06-01 04:21:41.137018] I [client-handshake.c:1636:select_server_supported_programs] 45-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:41.140117] I [client-handshake.c:1433:client_setvolume_cbk] 45-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:41.140139] I [client-handshake.c:1445:client_setvolume_cbk] 45-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:41.191542] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 45 >[2012-06-01 04:21:41.192949] I [client-handshake.c:453:client_set_lk_version_cbk] 45-master-client-3: Server lk version = 1 >[2012-06-01 04:21:41.200215] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 45-master-replicate-0: added root inode >[2012-06-01 04:21:41.205793] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 45-master-replicate-1: added root inode >[2012-06-01 04:21:41.230667] I [client.c:2151:notify] 44-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:41.230720] I [client.c:2151:notify] 44-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:41.230747] I [client.c:2151:notify] 44-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:41.230772] I [client.c:2151:notify] 44-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:41.234100] I [client.c:2090:client_rpc_notify] 44-master-client-0: disconnected >[2012-06-01 04:21:41.234134] I [client.c:2090:client_rpc_notify] 44-master-client-1: disconnected >[2012-06-01 04:21:41.234147] E [afr-common.c:3664:afr_notify] 44-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:41.234168] I [client.c:2090:client_rpc_notify] 44-master-client-2: disconnected >[2012-06-01 04:21:41.234677] I [client.c:2090:client_rpc_notify] 44-master-client-3: disconnected >[2012-06-01 04:21:41.234694] E [afr-common.c:3664:afr_notify] 44-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:45.567087] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:21:46.658736] I [io-cache.c:1549:check_cache_size_ok] 46-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:21:46.658811] I [io-cache.c:1549:check_cache_size_ok] 46-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:21:46.700798] I [client.c:2142:notify] 46-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:46.713654] I [client.c:2142:notify] 46-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:46.716364] I [client.c:2142:notify] 46-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:21:46.723926] I [client.c:2142:notify] 46-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:21:46.746385] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:21:46.746880] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:21:46.747108] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:21:46.747914] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 46-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:21:50.149578] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:50.151123] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:50.157343] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:21:50.157372] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:50.159043] I [afr-common.c:3627:afr_notify] 46-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:21:50.159420] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:50.159483] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:21:50.159499] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:50.163456] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-1: Server lk version = 1 >[2012-06-01 04:21:50.163494] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:21:50.163511] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:50.163681] I [afr-common.c:3627:afr_notify] 46-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:21:50.163713] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-0: Server lk version = 1 >[2012-06-01 04:21:50.165767] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-2: Server lk version = 1 >[2012-06-01 04:21:50.167902] I [client-handshake.c:1636:select_server_supported_programs] 46-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:21:50.169360] I [client-handshake.c:1433:client_setvolume_cbk] 46-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:21:50.169381] I [client-handshake.c:1445:client_setvolume_cbk] 46-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:21:50.196255] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 46 >[2012-06-01 04:21:50.200728] I [client-handshake.c:453:client_set_lk_version_cbk] 46-master-client-3: Server lk version = 1 >[2012-06-01 04:21:50.215713] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 46-master-replicate-0: added root inode >[2012-06-01 04:21:50.217188] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 46-master-replicate-1: added root inode >[2012-06-01 04:21:50.226915] W [quota.c:2377:quota_fsync_cbk] 45-master-quota: quota context not set in inode (gfid:99e7a5c0-6e3f-4b38-a81a-8e11cb242402) >[2012-06-01 04:21:50.274021] W [quota.c:2377:quota_fsync_cbk] 45-master-quota: quota context not set in inode (gfid:ae1bc032-d549-4d9c-a084-6d8750427624) >[2012-06-01 04:21:50.277566] I [client.c:2151:notify] 45-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:50.277644] I [client.c:2151:notify] 45-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:50.277678] I [client.c:2151:notify] 45-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:50.277710] I [client.c:2151:notify] 45-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:21:50.282164] I [client.c:2090:client_rpc_notify] 45-master-client-0: disconnected >[2012-06-01 04:21:50.282200] I [client.c:2090:client_rpc_notify] 45-master-client-1: disconnected >[2012-06-01 04:21:50.282212] E [afr-common.c:3664:afr_notify] 45-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:50.282233] I [client.c:2090:client_rpc_notify] 45-master-client-2: disconnected >[2012-06-01 04:21:50.282254] I [client.c:2090:client_rpc_notify] 45-master-client-3: disconnected >[2012-06-01 04:21:50.282265] E [afr-common.c:3664:afr_notify] 45-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:21:54.243496] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:05.812391] I [io-cache.c:1549:check_cache_size_ok] 47-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:05.812463] I [io-cache.c:1549:check_cache_size_ok] 47-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:05.812518] I [quota.c:3085:quota_parse_limits] 47-master-quota: no "limit-set" option provided >[2012-06-01 04:22:05.817725] I [client.c:2142:notify] 47-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:05.828587] I [client.c:2142:notify] 47-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:05.831151] I [client.c:2142:notify] 47-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:05.839937] I [client.c:2142:notify] 47-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:05.854559] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:05.854659] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:05.854736] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:05.854856] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 47-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:09.190722] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:09.191371] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:09.191546] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:09.191712] I [client-handshake.c:1636:select_server_supported_programs] 47-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:09.192286] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:09.192307] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:09.192421] I [afr-common.c:3627:afr_notify] 47-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:09.192457] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:09.192470] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:09.192834] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:09.192854] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:09.192991] I [afr-common.c:3627:afr_notify] 47-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:09.193026] I [client-handshake.c:1433:client_setvolume_cbk] 47-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:09.193047] I [client-handshake.c:1445:client_setvolume_cbk] 47-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:09.202624] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 47 >[2012-06-01 04:22:09.203140] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-0: Server lk version = 1 >[2012-06-01 04:22:09.203172] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-1: Server lk version = 1 >[2012-06-01 04:22:09.203193] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-2: Server lk version = 1 >[2012-06-01 04:22:09.203213] I [client-handshake.c:453:client_set_lk_version_cbk] 47-master-client-3: Server lk version = 1 >[2012-06-01 04:22:09.207689] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 47-master-replicate-0: added root inode >[2012-06-01 04:22:09.221711] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 47-master-replicate-1: added root inode >[2012-06-01 04:22:09.332959] I [client.c:2151:notify] 46-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:09.333029] I [client.c:2151:notify] 46-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:09.333060] I [client.c:2151:notify] 46-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:09.333086] I [client.c:2151:notify] 46-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:09.338281] I [client.c:2090:client_rpc_notify] 46-master-client-0: disconnected >[2012-06-01 04:22:09.338319] I [client.c:2090:client_rpc_notify] 46-master-client-1: disconnected >[2012-06-01 04:22:09.338332] E [afr-common.c:3664:afr_notify] 46-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:09.338354] I [client.c:2090:client_rpc_notify] 46-master-client-2: disconnected >[2012-06-01 04:22:09.338378] I [client.c:2090:client_rpc_notify] 46-master-client-3: disconnected >[2012-06-01 04:22:09.338387] E [afr-common.c:3664:afr_notify] 46-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:09.453100] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:10.499759] I [io-cache.c:1549:check_cache_size_ok] 48-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:10.499838] I [io-cache.c:1549:check_cache_size_ok] 48-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:10.507399] I [client.c:2142:notify] 48-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:10.511312] I [client.c:2142:notify] 48-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:10.514026] I [client.c:2142:notify] 48-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:10.516435] I [client.c:2142:notify] 48-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:10.523300] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:10.523415] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:10.523550] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:10.523649] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 48-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:13.304083] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:14.205212] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:14.205629] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:14.205655] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:14.205800] I [afr-common.c:3627:afr_notify] 48-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:14.205928] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-0: Server lk version = 1 >[2012-06-01 04:22:14.219049] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:14.219953] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:14.219980] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:14.222207] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-1: Server lk version = 1 >[2012-06-01 04:22:14.226570] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:14.227814] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:14.227847] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:14.228019] I [afr-common.c:3627:afr_notify] 48-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:14.229728] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-2: Server lk version = 1 >[2012-06-01 04:22:14.234203] I [client-handshake.c:1636:select_server_supported_programs] 48-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:14.234454] I [client-handshake.c:1433:client_setvolume_cbk] 48-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:14.234478] I [client-handshake.c:1445:client_setvolume_cbk] 48-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:14.241344] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 48 >[2012-06-01 04:22:14.241793] I [client-handshake.c:453:client_set_lk_version_cbk] 48-master-client-3: Server lk version = 1 >[2012-06-01 04:22:14.243819] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 48-master-replicate-0: added root inode >[2012-06-01 04:22:14.244830] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 48-master-replicate-1: added root inode >[2012-06-01 04:22:14.340120] W [quota.c:2377:quota_fsync_cbk] 47-master-quota: quota context not set in inode (gfid:a643fd8b-c2fd-4d3a-8dc6-efb7683c3cbb) >[2012-06-01 04:22:14.405808] W [quota.c:2377:quota_fsync_cbk] 47-master-quota: quota context not set in inode (gfid:9cd8b3d8-7e7a-45c0-8b3d-a2600e946953) >[2012-06-01 04:22:14.408276] I [client.c:2151:notify] 47-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:14.408341] I [client.c:2151:notify] 47-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:14.408377] I [client.c:2151:notify] 47-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:14.408408] I [client.c:2151:notify] 47-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:14.412441] I [client.c:2090:client_rpc_notify] 47-master-client-0: disconnected >[2012-06-01 04:22:14.412500] I [client.c:2090:client_rpc_notify] 47-master-client-1: disconnected >[2012-06-01 04:22:14.412530] E [afr-common.c:3664:afr_notify] 47-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:14.412564] I [client.c:2090:client_rpc_notify] 47-master-client-2: disconnected >[2012-06-01 04:22:14.412594] I [client.c:2090:client_rpc_notify] 47-master-client-3: disconnected >[2012-06-01 04:22:14.412609] E [afr-common.c:3664:afr_notify] 47-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:14.855986] I [io-cache.c:1549:check_cache_size_ok] 49-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:14.856128] I [io-cache.c:1549:check_cache_size_ok] 49-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:14.856193] I [quota.c:3085:quota_parse_limits] 49-master-quota: no "limit-set" option provided >[2012-06-01 04:22:14.866508] I [client.c:2142:notify] 49-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:14.871607] I [client.c:2142:notify] 49-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:14.876360] I [client.c:2142:notify] 49-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:14.882274] I [client.c:2142:notify] 49-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:14.896991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:14.897229] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:14.897349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:14.897466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 49-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:17.948743] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:18.247197] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:18.248985] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:18.249011] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:18.249149] I [afr-common.c:3627:afr_notify] 49-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:18.249864] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-0: Server lk version = 1 >[2012-06-01 04:22:18.249911] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:18.250921] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:18.250947] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:18.253553] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-1: Server lk version = 1 >[2012-06-01 04:22:18.261997] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:18.262227] I [client-handshake.c:1636:select_server_supported_programs] 49-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:18.263105] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:18.263130] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:18.263265] I [afr-common.c:3627:afr_notify] 49-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:18.263587] I [client-handshake.c:1433:client_setvolume_cbk] 49-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:18.263610] I [client-handshake.c:1445:client_setvolume_cbk] 49-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:18.272771] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 49 >[2012-06-01 04:22:18.273038] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-2: Server lk version = 1 >[2012-06-01 04:22:18.273072] I [client-handshake.c:453:client_set_lk_version_cbk] 49-master-client-3: Server lk version = 1 >[2012-06-01 04:22:18.276074] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 49-master-replicate-0: added root inode >[2012-06-01 04:22:18.277458] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 49-master-replicate-1: added root inode >[2012-06-01 04:22:18.335357] I [client.c:2151:notify] 48-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:18.335433] I [client.c:2151:notify] 48-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:18.335470] I [client.c:2151:notify] 48-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:18.335500] I [client.c:2151:notify] 48-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:18.336280] I [client.c:2090:client_rpc_notify] 48-master-client-0: disconnected >[2012-06-01 04:22:18.336371] I [client.c:2090:client_rpc_notify] 48-master-client-1: disconnected >[2012-06-01 04:22:18.336400] E [afr-common.c:3664:afr_notify] 48-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:18.336430] I [client.c:2090:client_rpc_notify] 48-master-client-2: disconnected >[2012-06-01 04:22:18.336459] I [client.c:2090:client_rpc_notify] 48-master-client-3: disconnected >[2012-06-01 04:22:18.336473] E [afr-common.c:3664:afr_notify] 48-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:19.081019] I [io-cache.c:1549:check_cache_size_ok] 50-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:19.081130] I [io-cache.c:1549:check_cache_size_ok] 50-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:19.171562] I [client.c:2142:notify] 50-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:19.182372] I [client.c:2142:notify] 50-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:19.187046] I [client.c:2142:notify] 50-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:19.205318] I [client.c:2142:notify] 50-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:19.215356] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:19.215504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:19.215666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:19.215785] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 50-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:21.401169] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:22.270596] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:22.270823] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:22.272315] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:22.272341] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:22.272485] I [afr-common.c:3627:afr_notify] 50-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:22:22.273080] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:22.273100] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:22.273930] I [afr-common.c:3627:afr_notify] 50-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:22.274323] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:22.275594] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-1: Server lk version = 1 >[2012-06-01 04:22:22.275716] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-2: Server lk version = 1 >[2012-06-01 04:22:22.276638] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:22.276659] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:22.279138] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-0: Server lk version = 1 >[2012-06-01 04:22:22.291120] I [client-handshake.c:1636:select_server_supported_programs] 50-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:22.292076] I [client-handshake.c:1433:client_setvolume_cbk] 50-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:22.292097] I [client-handshake.c:1445:client_setvolume_cbk] 50-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:22.315236] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 50 >[2012-06-01 04:22:22.316248] I [client-handshake.c:453:client_set_lk_version_cbk] 50-master-client-3: Server lk version = 1 >[2012-06-01 04:22:22.325531] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 50-master-replicate-0: added root inode >[2012-06-01 04:22:22.326907] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 50-master-replicate-1: added root inode >[2012-06-01 04:22:22.498424] I [client.c:2151:notify] 49-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:22.498508] I [client.c:2151:notify] 49-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:22.498554] I [client.c:2151:notify] 49-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:22.498595] I [client.c:2151:notify] 49-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:22.503137] I [client.c:2090:client_rpc_notify] 49-master-client-0: disconnected >[2012-06-01 04:22:22.503196] I [client.c:2090:client_rpc_notify] 49-master-client-1: disconnected >[2012-06-01 04:22:22.503217] E [afr-common.c:3664:afr_notify] 49-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:22.503256] I [client.c:2090:client_rpc_notify] 49-master-client-2: disconnected >[2012-06-01 04:22:22.503294] I [client.c:2090:client_rpc_notify] 49-master-client-3: disconnected >[2012-06-01 04:22:22.503313] E [afr-common.c:3664:afr_notify] 49-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:22.771824] I [io-cache.c:1549:check_cache_size_ok] 51-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:22.771953] I [io-cache.c:1549:check_cache_size_ok] 51-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:22.772053] I [quota.c:3085:quota_parse_limits] 51-master-quota: no "limit-set" option provided >[2012-06-01 04:22:22.784070] I [client.c:2142:notify] 51-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:22.794895] I [client.c:2142:notify] 51-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:22.809660] I [client.c:2142:notify] 51-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:22.821020] I [client.c:2142:notify] 51-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:22.894579] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:22.897524] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:22.898234] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:22.898666] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 51-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:26.299825] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:26.300265] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:26.300765] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:26.300787] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:26.301254] I [afr-common.c:3627:afr_notify] 51-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:26.301294] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:26.301309] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:26.306631] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-0: Server lk version = 1 >[2012-06-01 04:22:26.307362] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-1: Server lk version = 1 >[2012-06-01 04:22:26.322418] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:26.323784] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:26.324410] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:26.325686] I [afr-common.c:3627:afr_notify] 51-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:26.327858] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-2: Server lk version = 1 >[2012-06-01 04:22:26.340816] I [client-handshake.c:1636:select_server_supported_programs] 51-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:26.343182] I [client-handshake.c:1433:client_setvolume_cbk] 51-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:26.343228] I [client-handshake.c:1445:client_setvolume_cbk] 51-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:26.353854] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 51 >[2012-06-01 04:22:26.355004] I [client-handshake.c:453:client_set_lk_version_cbk] 51-master-client-3: Server lk version = 1 >[2012-06-01 04:22:26.362228] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 51-master-replicate-0: added root inode >[2012-06-01 04:22:26.365429] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 51-master-replicate-1: added root inode >[2012-06-01 04:22:26.390160] I [client.c:2151:notify] 50-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:26.390226] I [client.c:2151:notify] 50-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:26.390260] I [client.c:2151:notify] 50-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:26.390288] I [client.c:2151:notify] 50-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:26.397075] I [client.c:2090:client_rpc_notify] 50-master-client-0: disconnected >[2012-06-01 04:22:26.397115] I [client.c:2090:client_rpc_notify] 50-master-client-1: disconnected >[2012-06-01 04:22:26.397130] E [afr-common.c:3664:afr_notify] 50-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:26.397164] I [client.c:2090:client_rpc_notify] 50-master-client-2: disconnected >[2012-06-01 04:22:26.397190] I [client.c:2090:client_rpc_notify] 50-master-client-3: disconnected >[2012-06-01 04:22:26.397202] E [afr-common.c:3664:afr_notify] 50-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:30.958870] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:32.145307] I [io-cache.c:1549:check_cache_size_ok] 52-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:32.145378] I [io-cache.c:1549:check_cache_size_ok] 52-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:32.208777] I [client.c:2142:notify] 52-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:32.211943] I [client.c:2142:notify] 52-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:32.217839] I [client.c:2142:notify] 52-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:32.223433] I [client.c:2142:notify] 52-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:32.231450] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:32.231652] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:32.231814] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:32.231991] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 52-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:35.352501] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:35.353443] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:35.353465] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:35.353583] I [afr-common.c:3627:afr_notify] 52-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:35.355963] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-0: Server lk version = 1 >[2012-06-01 04:22:35.356507] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:35.357004] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:35.357023] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:35.357339] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-1: Server lk version = 1 >[2012-06-01 04:22:35.382089] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:35.383645] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:35.383666] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:35.384606] I [afr-common.c:3627:afr_notify] 52-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:35.385504] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-2: Server lk version = 1 >[2012-06-01 04:22:35.393689] I [client-handshake.c:1636:select_server_supported_programs] 52-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:35.395756] I [client-handshake.c:1433:client_setvolume_cbk] 52-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:35.395779] I [client-handshake.c:1445:client_setvolume_cbk] 52-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:35.428479] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 52 >[2012-06-01 04:22:35.428781] I [client-handshake.c:453:client_set_lk_version_cbk] 52-master-client-3: Server lk version = 1 >[2012-06-01 04:22:35.436528] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 52-master-replicate-0: added root inode >[2012-06-01 04:22:35.437928] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 52-master-replicate-1: added root inode >[2012-06-01 04:22:35.462261] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:a43e29f2-2d14-4191-8ee7-0507690c8b76) >[2012-06-01 04:22:35.468760] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b) >[2012-06-01 04:22:35.479247] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:05d26344-33d0-4f42-80d6-a3dd2695fb2b) >[2012-06-01 04:22:35.487828] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:561277dc-03e2-4127-baee-b556fc9799f3) >[2012-06-01 04:22:35.492668] W [quota.c:2377:quota_fsync_cbk] 51-master-quota: quota context not set in inode (gfid:81f303f5-e0c4-419a-8cb5-a808cbd1015e) >[2012-06-01 04:22:35.495528] I [client.c:2151:notify] 51-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:35.495585] I [client.c:2151:notify] 51-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:35.495618] I [client.c:2151:notify] 51-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:35.495643] I [client.c:2151:notify] 51-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:35.499592] I [client.c:2090:client_rpc_notify] 51-master-client-0: disconnected >[2012-06-01 04:22:35.499630] I [client.c:2090:client_rpc_notify] 51-master-client-1: disconnected >[2012-06-01 04:22:35.499644] E [afr-common.c:3664:afr_notify] 51-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:35.499668] I [client.c:2090:client_rpc_notify] 51-master-client-2: disconnected >[2012-06-01 04:22:35.499690] I [client.c:2090:client_rpc_notify] 51-master-client-3: disconnected >[2012-06-01 04:22:35.499719] E [afr-common.c:3664:afr_notify] 51-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:35.625775] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:37.070914] I [io-cache.c:1549:check_cache_size_ok] 53-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:37.071040] I [io-cache.c:1549:check_cache_size_ok] 53-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:37.071142] I [quota.c:3085:quota_parse_limits] 53-master-quota: no "limit-set" option provided >[2012-06-01 04:22:37.227877] I [client.c:2142:notify] 53-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:37.237168] I [client.c:2142:notify] 53-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:37.248704] I [client.c:2142:notify] 53-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:37.276149] I [client.c:2142:notify] 53-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:37.323031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:37.323337] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:37.323552] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:37.323712] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 53-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:40.404417] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:40.408601] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:40.408663] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:40.408848] I [afr-common.c:3627:afr_notify] 53-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:40.409353] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-0: Server lk version = 1 >[2012-06-01 04:22:40.427523] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:40.435910] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:40.435946] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:40.437732] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-1: Server lk version = 1 >[2012-06-01 04:22:40.458480] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:40.460446] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:40.460480] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:40.460718] I [afr-common.c:3627:afr_notify] 53-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:40.462492] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-2: Server lk version = 1 >[2012-06-01 04:22:40.488228] I [client-handshake.c:1636:select_server_supported_programs] 53-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:40.491173] I [client-handshake.c:1433:client_setvolume_cbk] 53-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:40.491208] I [client-handshake.c:1445:client_setvolume_cbk] 53-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:40.524895] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 53 >[2012-06-01 04:22:40.531081] I [client-handshake.c:453:client_set_lk_version_cbk] 53-master-client-3: Server lk version = 1 >[2012-06-01 04:22:40.538997] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 53-master-replicate-0: added root inode >[2012-06-01 04:22:40.541472] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 53-master-replicate-1: added root inode >[2012-06-01 04:22:40.602319] I [client.c:2151:notify] 52-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:40.602394] I [client.c:2151:notify] 52-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:40.602437] I [client.c:2151:notify] 52-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:40.602478] I [client.c:2151:notify] 52-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:40.609290] I [client.c:2090:client_rpc_notify] 52-master-client-0: disconnected >[2012-06-01 04:22:40.609332] I [client.c:2090:client_rpc_notify] 52-master-client-1: disconnected >[2012-06-01 04:22:40.609348] E [afr-common.c:3664:afr_notify] 52-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:40.610214] I [client.c:2090:client_rpc_notify] 52-master-client-2: disconnected >[2012-06-01 04:22:40.610247] I [client.c:2090:client_rpc_notify] 52-master-client-3: disconnected >[2012-06-01 04:22:40.610259] E [afr-common.c:3664:afr_notify] 52-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:40.981009] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:47.321224] I [io-cache.c:1549:check_cache_size_ok] 54-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:47.321355] I [io-cache.c:1549:check_cache_size_ok] 54-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:47.416007] I [client.c:2142:notify] 54-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:47.454686] I [client.c:2142:notify] 54-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:47.460190] I [client.c:2142:notify] 54-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:47.462666] I [client.c:2142:notify] 54-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:47.477502] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:47.477922] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:47.484022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:47.484643] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 54-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:50.520752] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:50.539139] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:50.540479] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:50.540509] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:50.540583] I [afr-common.c:3627:afr_notify] 54-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:50.542445] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:50.542468] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:50.544903] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-1: Server lk version = 1 >[2012-06-01 04:22:50.544937] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-0: Server lk version = 1 >[2012-06-01 04:22:51.544810] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:51.546671] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:51.546694] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:51.546815] I [afr-common.c:3627:afr_notify] 54-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:22:51.546849] I [client-handshake.c:1636:select_server_supported_programs] 54-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:51.548579] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-2: Server lk version = 1 >[2012-06-01 04:22:51.548619] I [client-handshake.c:1433:client_setvolume_cbk] 54-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:51.548629] I [client-handshake.c:1445:client_setvolume_cbk] 54-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:51.578266] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 54 >[2012-06-01 04:22:51.578558] I [client-handshake.c:453:client_set_lk_version_cbk] 54-master-client-3: Server lk version = 1 >[2012-06-01 04:22:51.580395] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 54-master-replicate-0: added root inode >[2012-06-01 04:22:51.581802] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 54-master-replicate-1: added root inode >[2012-06-01 04:22:51.592892] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:23b92615-8b00-4f7d-919b-dd30d836b438) >[2012-06-01 04:22:51.646954] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:09b17a28-6b3d-4fe1-bb09-e7caefdef1ce) >[2012-06-01 04:22:51.654440] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:d55db600-df01-40a6-852b-f66cd5332de7) >[2012-06-01 04:22:51.669529] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:161164b1-58e3-4eef-9864-0ffcb97521c0) >[2012-06-01 04:22:51.696840] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:8bab20b7-696b-4ba6-91c6-68c7ddd96714) >[2012-06-01 04:22:51.700746] W [quota.c:2377:quota_fsync_cbk] 53-master-quota: quota context not set in inode (gfid:c906f6b4-7e84-4ac5-a6f0-536b45dd9b4f) >[2012-06-01 04:22:51.710118] I [client.c:2151:notify] 53-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:51.710249] I [client.c:2151:notify] 53-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:51.710293] I [client.c:2151:notify] 53-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:51.710324] I [client.c:2151:notify] 53-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:51.727570] I [client.c:2090:client_rpc_notify] 53-master-client-0: disconnected >[2012-06-01 04:22:51.727638] I [client.c:2090:client_rpc_notify] 53-master-client-1: disconnected >[2012-06-01 04:22:51.727656] E [afr-common.c:3664:afr_notify] 53-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:51.727872] I [client.c:2090:client_rpc_notify] 53-master-client-2: disconnected >[2012-06-01 04:22:51.727914] I [client.c:2090:client_rpc_notify] 53-master-client-3: disconnected >[2012-06-01 04:22:51.727927] E [afr-common.c:3664:afr_notify] 53-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:52.860325] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:22:54.702128] I [io-cache.c:1549:check_cache_size_ok] 55-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:22:54.702231] I [io-cache.c:1549:check_cache_size_ok] 55-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:22:54.702319] I [quota.c:3085:quota_parse_limits] 55-master-quota: no "limit-set" option provided >[2012-06-01 04:22:55.008186] I [client.c:2142:notify] 55-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:55.036223] I [client.c:2142:notify] 55-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:55.052030] I [client.c:2142:notify] 55-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:22:55.061427] I [client.c:2142:notify] 55-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:22:55.116773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:22:55.117016] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:22:55.117142] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:22:55.117271] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 55-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:22:58.549959] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:58.550663] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:22:58.550680] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:58.550801] I [afr-common.c:3627:afr_notify] 55-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:22:58.551134] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-0: Server lk version = 1 >[2012-06-01 04:22:58.551172] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:58.557787] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:22:58.557798] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:58.557944] I [afr-common.c:3627:afr_notify] 55-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:22:58.558660] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-3: Server lk version = 1 >[2012-06-01 04:22:58.593402] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:58.593905] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:22:58.593923] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:58.594955] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-1: Server lk version = 1 >[2012-06-01 04:22:58.620628] I [client-handshake.c:1636:select_server_supported_programs] 55-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:22:58.633137] I [client-handshake.c:1433:client_setvolume_cbk] 55-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:22:58.633166] I [client-handshake.c:1445:client_setvolume_cbk] 55-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:22:58.741674] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 55 >[2012-06-01 04:22:58.742530] I [client-handshake.c:453:client_set_lk_version_cbk] 55-master-client-2: Server lk version = 1 >[2012-06-01 04:22:58.750076] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 55-master-replicate-0: added root inode >[2012-06-01 04:22:58.751027] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 55-master-replicate-1: added root inode >[2012-06-01 04:22:59.039433] I [client.c:2151:notify] 54-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:59.039545] I [client.c:2151:notify] 54-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:59.039612] I [client.c:2151:notify] 54-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:59.039684] I [client.c:2151:notify] 54-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:22:59.135979] I [client.c:2090:client_rpc_notify] 54-master-client-0: disconnected >[2012-06-01 04:22:59.136680] I [client.c:2090:client_rpc_notify] 54-master-client-1: disconnected >[2012-06-01 04:22:59.139826] E [afr-common.c:3664:afr_notify] 54-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:22:59.139892] I [client.c:2090:client_rpc_notify] 54-master-client-2: disconnected >[2012-06-01 04:22:59.142019] I [client.c:2090:client_rpc_notify] 54-master-client-3: disconnected >[2012-06-01 04:22:59.142049] E [afr-common.c:3664:afr_notify] 54-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:00.275492] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:01.687540] I [io-cache.c:1549:check_cache_size_ok] 56-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:01.687654] I [io-cache.c:1549:check_cache_size_ok] 56-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:01.701416] I [client.c:2142:notify] 56-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:01.708544] I [client.c:2142:notify] 56-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:01.714124] I [client.c:2142:notify] 56-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:01.718149] I [client.c:2142:notify] 56-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:01.732764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:01.732893] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:01.733166] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:01.733253] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 56-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:05.637029] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:05.641574] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:05.641960] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:05.641981] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:05.642047] I [afr-common.c:3627:afr_notify] 56-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:05.642080] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:05.642093] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:05.642320] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-0: Server lk version = 1 >[2012-06-01 04:23:05.642349] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-1: Server lk version = 1 >[2012-06-01 04:23:05.642367] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:05.642808] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:05.642835] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:05.642879] I [afr-common.c:3627:afr_notify] 56-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:05.644162] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-2: Server lk version = 1 >[2012-06-01 04:23:05.670760] I [client-handshake.c:1636:select_server_supported_programs] 56-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:05.672346] I [client-handshake.c:1433:client_setvolume_cbk] 56-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:05.672368] I [client-handshake.c:1445:client_setvolume_cbk] 56-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:05.679819] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 56 >[2012-06-01 04:23:05.683623] I [client-handshake.c:453:client_set_lk_version_cbk] 56-master-client-3: Server lk version = 1 >[2012-06-01 04:23:05.685037] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 56-master-replicate-0: added root inode >[2012-06-01 04:23:05.691379] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 56-master-replicate-1: added root inode >[2012-06-01 04:23:05.692820] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:05.695112] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:a97a35c3-a7aa-4b3c-b400-50164d53446f) >[2012-06-01 04:23:05.793497] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:0abe7955-b748-4fc7-9f26-1e823949458b) >[2012-06-01 04:23:05.796189] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:b2374a2f-cfed-4df8-a691-ace626e2e6b4) >[2012-06-01 04:23:05.811259] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:d0e3460f-c940-4dc1-b991-839d48e3965e) >[2012-06-01 04:23:05.825710] W [quota.c:2377:quota_fsync_cbk] 55-master-quota: quota context not set in inode (gfid:debaad44-cdff-483e-b38c-6d04a346efb9) >[2012-06-01 04:23:05.836399] I [client.c:2151:notify] 55-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:05.836473] I [client.c:2151:notify] 55-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:05.836514] I [client.c:2151:notify] 55-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:05.836552] I [client.c:2151:notify] 55-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:05.856491] I [client.c:2090:client_rpc_notify] 55-master-client-0: disconnected >[2012-06-01 04:23:05.856536] I [client.c:2090:client_rpc_notify] 55-master-client-1: disconnected >[2012-06-01 04:23:05.856551] E [afr-common.c:3664:afr_notify] 55-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:05.856580] I [client.c:2090:client_rpc_notify] 55-master-client-2: disconnected >[2012-06-01 04:23:05.856606] I [client.c:2090:client_rpc_notify] 55-master-client-3: disconnected >[2012-06-01 04:23:05.856644] E [afr-common.c:3664:afr_notify] 55-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:07.039103] I [io-cache.c:1549:check_cache_size_ok] 57-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:07.039196] I [io-cache.c:1549:check_cache_size_ok] 57-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:07.039267] I [quota.c:3085:quota_parse_limits] 57-master-quota: no "limit-set" option provided >[2012-06-01 04:23:07.113931] I [client.c:2142:notify] 57-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:07.134929] I [client.c:2142:notify] 57-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:07.142599] I [client.c:2142:notify] 57-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:07.153168] I [client.c:2142:notify] 57-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:07.167989] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:07.169500] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:07.169600] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:07.169676] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 57-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:10.115128] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:10.679050] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:10.679142] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:10.684360] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:10.684383] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:10.684522] I [afr-common.c:3627:afr_notify] 57-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:10.684559] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:10.684572] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:10.684692] I [afr-common.c:3627:afr_notify] 57-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:10.685009] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-0: Server lk version = 1 >[2012-06-01 04:23:10.685045] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-2: Server lk version = 1 >[2012-06-01 04:23:10.685376] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:10.687190] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:10.687212] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:10.689660] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-3: Server lk version = 1 >[2012-06-01 04:23:10.726906] I [client-handshake.c:1636:select_server_supported_programs] 57-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:10.733457] I [client-handshake.c:1433:client_setvolume_cbk] 57-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:10.736803] I [client-handshake.c:1445:client_setvolume_cbk] 57-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:10.758662] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 57 >[2012-06-01 04:23:10.759406] I [client-handshake.c:453:client_set_lk_version_cbk] 57-master-client-1: Server lk version = 1 >[2012-06-01 04:23:10.759655] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 57-master-replicate-0: added root inode >[2012-06-01 04:23:10.764060] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 57-master-replicate-1: added root inode >[2012-06-01 04:23:10.959817] I [client.c:2151:notify] 56-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:10.959920] I [client.c:2151:notify] 56-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:10.959975] I [client.c:2151:notify] 56-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:10.960020] I [client.c:2151:notify] 56-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:10.974253] I [client.c:2090:client_rpc_notify] 56-master-client-0: disconnected >[2012-06-01 04:23:10.974292] I [client.c:2090:client_rpc_notify] 56-master-client-1: disconnected >[2012-06-01 04:23:10.974304] E [afr-common.c:3664:afr_notify] 56-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:10.974330] I [client.c:2090:client_rpc_notify] 56-master-client-2: disconnected >[2012-06-01 04:23:10.974354] I [client.c:2090:client_rpc_notify] 56-master-client-3: disconnected >[2012-06-01 04:23:10.974367] E [afr-common.c:3664:afr_notify] 56-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:11.198235] I [io-cache.c:1549:check_cache_size_ok] 58-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:11.198336] I [io-cache.c:1549:check_cache_size_ok] 58-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:11.449432] I [client.c:2142:notify] 58-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:11.485453] I [client.c:2142:notify] 58-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:11.500915] I [client.c:2142:notify] 58-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:11.513585] I [client.c:2142:notify] 58-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:11.533803] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:11.534197] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:11.534276] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:11.534356] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 58-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:14.750121] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:14.752597] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:14.752693] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:14.753702] I [afr-common.c:3627:afr_notify] 58-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:14.753758] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:14.765873] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-0: Server lk version = 1 >[2012-06-01 04:23:14.765929] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:14.765952] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:14.770042] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-1: Server lk version = 1 >[2012-06-01 04:23:14.781016] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:14.789010] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:14.791539] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:14.791792] I [afr-common.c:3627:afr_notify] 58-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:14.794133] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-2: Server lk version = 1 >[2012-06-01 04:23:14.794184] I [client-handshake.c:1636:select_server_supported_programs] 58-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:14.803123] I [client-handshake.c:1433:client_setvolume_cbk] 58-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:14.803204] I [client-handshake.c:1445:client_setvolume_cbk] 58-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:14.815822] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 58 >[2012-06-01 04:23:14.820484] I [client-handshake.c:453:client_set_lk_version_cbk] 58-master-client-3: Server lk version = 1 >[2012-06-01 04:23:14.824728] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 58-master-replicate-0: added root inode >[2012-06-01 04:23:14.827545] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 58-master-replicate-1: added root inode >[2012-06-01 04:23:14.905884] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:9e934735-4d2d-48e5-b1e5-2af76431ab24) >[2012-06-01 04:23:14.933230] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:948ce967-4e02-4ce0-aef8-a10ffd7ab165) >[2012-06-01 04:23:14.939682] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:ffeac997-bf44-4a78-aed1-142f2cd9c1f4) >[2012-06-01 04:23:14.961065] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:df424240-d25c-44cb-96fe-f87e1c7fb2ab) >[2012-06-01 04:23:15.009204] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:caa8bd7b-2bd7-4dd2-b5c3-4517a80b5be8) >[2012-06-01 04:23:15.031164] W [quota.c:2377:quota_fsync_cbk] 57-master-quota: quota context not set in inode (gfid:94bc5eda-6b69-40c0-94fc-6f95c47a7693) >[2012-06-01 04:23:15.042789] I [client.c:2151:notify] 57-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:15.042842] I [client.c:2151:notify] 57-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:15.042905] I [client.c:2151:notify] 57-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:15.042947] I [client.c:2151:notify] 57-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:23:15.066580] I [client.c:2090:client_rpc_notify] 57-master-client-0: disconnected >[2012-06-01 04:23:15.066624] I [client.c:2090:client_rpc_notify] 57-master-client-1: disconnected >[2012-06-01 04:23:15.066637] E [afr-common.c:3664:afr_notify] 57-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:15.066664] I [client.c:2090:client_rpc_notify] 57-master-client-2: disconnected >[2012-06-01 04:23:15.066692] I [client.c:2090:client_rpc_notify] 57-master-client-3: disconnected >[2012-06-01 04:23:15.066703] E [afr-common.c:3664:afr_notify] 57-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:23:15.070649] W [fuse-resolve.c:359:fuse_resolve_fd] 0-fuse-resolve: migration of fd (0x1b1bce94) did not complete, failing fop with EBADF >[2012-06-01 04:23:15.074827] W [fuse-resolve.c:359:fuse_resolve_fd] 0-fuse-resolve: migration of fd (0x1b1bce94) did not complete, failing fop with EBADF >[2012-06-01 04:23:18.953460] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:20.177752] I [io-cache.c:1549:check_cache_size_ok] 59-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:20.177846] I [io-cache.c:1549:check_cache_size_ok] 59-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:20.191406] I [quota.c:3085:quota_parse_limits] 59-master-quota: no "limit-set" option provided >[2012-06-01 04:23:20.325798] I [client.c:2142:notify] 59-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:20.348541] I [client.c:2142:notify] 59-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:20.361327] I [client.c:2142:notify] 59-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:20.378620] I [client.c:2142:notify] 59-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:20.398119] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:20.398243] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:20.398349] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:20.398452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 59-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:22.800525] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:23.787160] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:23.787369] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:23.787389] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:23.787434] I [afr-common.c:3627:afr_notify] 59-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:23.787507] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-0: Server lk version = 1 >[2012-06-01 04:23:23.790686] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:23.791231] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:23.791255] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:23.791563] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-1: Server lk version = 1 >[2012-06-01 04:23:23.796046] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:23.796218] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:23.796236] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:23.796279] I [afr-common.c:3627:afr_notify] 59-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:23.796589] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-2: Server lk version = 1 >[2012-06-01 04:23:23.802313] I [client-handshake.c:1636:select_server_supported_programs] 59-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:23.802484] I [client-handshake.c:1433:client_setvolume_cbk] 59-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:23.802503] I [client-handshake.c:1445:client_setvolume_cbk] 59-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:23.807233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 59 >[2012-06-01 04:23:23.807303] I [client-handshake.c:453:client_set_lk_version_cbk] 59-master-client-3: Server lk version = 1 >[2012-06-01 04:23:23.858155] I [io-cache.c:1549:check_cache_size_ok] 60-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:23.858232] I [io-cache.c:1549:check_cache_size_ok] 60-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:23.927219] I [client.c:2142:notify] 60-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:23.939294] I [client.c:2142:notify] 60-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:23.961461] I [client.c:2142:notify] 60-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:23.964976] I [client.c:2142:notify] 60-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:23.993186] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:23.993236] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:23.993291] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:23.993326] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 60-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:27.066517] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:27.818374] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:27.819242] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:27.819277] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:27.819356] I [afr-common.c:3627:afr_notify] 60-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:27.819494] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-0: Server lk version = 1 >[2012-06-01 04:23:27.838318] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:27.838592] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:27.838651] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:27.839059] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-1: Server lk version = 1 >[2012-06-01 04:23:27.889598] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:27.891685] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:27.891716] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:27.891787] I [afr-common.c:3627:afr_notify] 60-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:27.892342] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-2: Server lk version = 1 >[2012-06-01 04:23:27.896011] I [client-handshake.c:1636:select_server_supported_programs] 60-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:27.896441] I [client-handshake.c:1433:client_setvolume_cbk] 60-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:27.896474] I [client-handshake.c:1445:client_setvolume_cbk] 60-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:27.917302] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 60 >[2012-06-01 04:23:27.917472] I [client-handshake.c:453:client_set_lk_version_cbk] 60-master-client-3: Server lk version = 1 >[2012-06-01 04:23:28.336396] I [io-cache.c:1549:check_cache_size_ok] 61-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:28.336469] I [io-cache.c:1549:check_cache_size_ok] 61-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:28.336541] I [quota.c:3085:quota_parse_limits] 61-master-quota: no "limit-set" option provided >[2012-06-01 04:23:28.440919] I [client.c:2142:notify] 61-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:28.449815] I [client.c:2142:notify] 61-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:28.485291] I [client.c:2142:notify] 61-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:28.500472] I [client.c:2142:notify] 61-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:28.590237] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:28.590357] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:28.598415] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:28.598942] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 61-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:31.935489] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:31.936002] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:31.936094] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:31.936700] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:31.936735] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:31.936845] I [afr-common.c:3627:afr_notify] 61-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:31.936900] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:31.936923] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:31.936996] I [afr-common.c:3627:afr_notify] 61-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:31.937046] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:31.937067] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:31.937230] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-0: Server lk version = 1 >[2012-06-01 04:23:31.937272] I [client-handshake.c:1636:select_server_supported_programs] 61-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:31.937379] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-1: Server lk version = 1 >[2012-06-01 04:23:31.938687] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-2: Server lk version = 1 >[2012-06-01 04:23:31.939049] I [client-handshake.c:1433:client_setvolume_cbk] 61-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:31.939084] I [client-handshake.c:1445:client_setvolume_cbk] 61-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:31.955826] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 61 >[2012-06-01 04:23:31.955969] I [client-handshake.c:453:client_set_lk_version_cbk] 61-master-client-3: Server lk version = 1 >[2012-06-01 04:23:33.076297] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:34.369814] I [io-cache.c:1549:check_cache_size_ok] 62-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:34.370435] I [io-cache.c:1549:check_cache_size_ok] 62-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:34.396917] I [client.c:2142:notify] 62-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:34.431748] I [client.c:2142:notify] 62-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:34.436939] I [client.c:2142:notify] 62-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:34.450319] I [client.c:2142:notify] 62-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:34.458156] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:34.458291] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:34.458411] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:34.458527] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 62-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:37.927490] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:37.933271] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:37.939296] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:37.944405] I [client-handshake.c:1636:select_server_supported_programs] 62-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:37.953615] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:37.953650] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:37.953729] I [afr-common.c:3627:afr_notify] 62-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:23:37.953782] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:37.953804] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:37.954074] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:37.954105] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:37.954180] I [afr-common.c:3627:afr_notify] 62-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:37.954230] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-1: Server lk version = 1 >[2012-06-01 04:23:37.954263] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-0: Server lk version = 1 >[2012-06-01 04:23:37.954943] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-2: Server lk version = 1 >[2012-06-01 04:23:37.954997] I [client-handshake.c:1433:client_setvolume_cbk] 62-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:37.955019] I [client-handshake.c:1445:client_setvolume_cbk] 62-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:37.961474] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 62 >[2012-06-01 04:23:37.961605] I [client-handshake.c:453:client_set_lk_version_cbk] 62-master-client-3: Server lk version = 1 >[2012-06-01 04:23:38.650978] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:39.916817] I [io-cache.c:1549:check_cache_size_ok] 63-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:39.917680] I [io-cache.c:1549:check_cache_size_ok] 63-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:39.917788] I [quota.c:3085:quota_parse_limits] 63-master-quota: no "limit-set" option provided >[2012-06-01 04:23:39.925727] I [client.c:2142:notify] 63-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:39.931935] I [client.c:2142:notify] 63-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:39.945128] I [client.c:2142:notify] 63-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:39.950025] I [client.c:2142:notify] 63-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:39.972067] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:39.972219] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:39.972491] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:39.973452] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 63-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:42.926305] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:43.980124] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:43.980415] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:43.980445] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:43.980533] I [afr-common.c:3627:afr_notify] 63-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:43.980667] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-0: Server lk version = 1 >[2012-06-01 04:23:44.005839] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:44.006176] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:44.006215] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:44.006509] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-1: Server lk version = 1 >[2012-06-01 04:23:44.046779] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:44.047106] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:44.047135] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:44.047208] I [afr-common.c:3627:afr_notify] 63-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:44.047793] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-2: Server lk version = 1 >[2012-06-01 04:23:44.063726] I [client-handshake.c:1636:select_server_supported_programs] 63-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:44.064032] I [client-handshake.c:1433:client_setvolume_cbk] 63-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:44.064061] I [client-handshake.c:1445:client_setvolume_cbk] 63-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:44.430936] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 63 >[2012-06-01 04:23:44.439518] I [client-handshake.c:453:client_set_lk_version_cbk] 63-master-client-3: Server lk version = 1 >[2012-06-01 04:23:44.451653] I [io-cache.c:1549:check_cache_size_ok] 64-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:44.451772] I [io-cache.c:1549:check_cache_size_ok] 64-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:44.512719] I [client.c:2142:notify] 64-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:44.530315] I [client.c:2142:notify] 64-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:44.546265] I [client.c:2142:notify] 64-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:44.549638] I [client.c:2142:notify] 64-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:44.558393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:44.567135] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:44.567226] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:44.567498] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 64-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:46.759311] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:47.952788] I [io-cache.c:1549:check_cache_size_ok] 65-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:47.952882] I [io-cache.c:1549:check_cache_size_ok] 65-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:47.952946] I [quota.c:3085:quota_parse_limits] 65-master-quota: no "limit-set" option provided >[2012-06-01 04:23:47.978703] I [client.c:2142:notify] 65-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:47.983081] I [client.c:2142:notify] 65-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:48.011713] I [client.c:2142:notify] 65-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:48.015011] I [client.c:2142:notify] 65-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:48.036031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:48.036115] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:48.036153] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:48.036234] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 65-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:48.114377] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:48.114573] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:48.114592] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:48.114645] I [afr-common.c:3627:afr_notify] 64-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:48.114736] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-0: Server lk version = 1 >[2012-06-01 04:23:48.144354] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:48.144562] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:48.144581] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:48.144797] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-1: Server lk version = 1 >[2012-06-01 04:23:48.166150] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:48.166348] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:48.166371] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:48.166425] I [afr-common.c:3627:afr_notify] 64-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:48.166523] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-2: Server lk version = 1 >[2012-06-01 04:23:48.169741] I [client-handshake.c:1636:select_server_supported_programs] 64-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:48.170021] I [client-handshake.c:1433:client_setvolume_cbk] 64-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:48.170040] I [client-handshake.c:1445:client_setvolume_cbk] 64-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:48.184018] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 64 >[2012-06-01 04:23:48.184136] I [client-handshake.c:453:client_set_lk_version_cbk] 64-master-client-3: Server lk version = 1 >[2012-06-01 04:23:50.761456] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:23:51.193536] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:51.193772] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:51.193793] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:51.193840] I [afr-common.c:3627:afr_notify] 65-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:51.194541] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-0: Server lk version = 1 >[2012-06-01 04:23:51.199293] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:51.199558] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:51.199578] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:51.199946] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-1: Server lk version = 1 >[2012-06-01 04:23:51.206187] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:51.206377] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:51.206395] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:51.206440] I [afr-common.c:3627:afr_notify] 65-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:51.206520] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-2: Server lk version = 1 >[2012-06-01 04:23:51.219464] I [client-handshake.c:1636:select_server_supported_programs] 65-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:51.219911] I [client-handshake.c:1433:client_setvolume_cbk] 65-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:51.219933] I [client-handshake.c:1445:client_setvolume_cbk] 65-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:51.224259] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 65 >[2012-06-01 04:23:51.224332] I [client-handshake.c:453:client_set_lk_version_cbk] 65-master-client-3: Server lk version = 1 >[2012-06-01 04:23:51.823547] I [io-cache.c:1549:check_cache_size_ok] 66-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:23:51.823644] I [io-cache.c:1549:check_cache_size_ok] 66-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:23:51.854023] I [client.c:2142:notify] 66-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:51.863120] I [client.c:2142:notify] 66-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:51.865895] I [client.c:2142:notify] 66-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:23:51.872757] I [client.c:2142:notify] 66-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:23:51.879482] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:23:51.879573] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:23:51.879668] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:23:51.879752] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 66-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:23:55.226793] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:55.227425] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:23:55.227447] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:55.227506] I [afr-common.c:3627:afr_notify] 66-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:23:55.227767] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-0: Server lk version = 1 >[2012-06-01 04:23:55.234619] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:55.234831] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:23:55.234852] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:55.234988] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-1: Server lk version = 1 >[2012-06-01 04:23:55.245157] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:55.245337] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:23:55.245354] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:55.245406] I [afr-common.c:3627:afr_notify] 66-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:23:55.245490] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-2: Server lk version = 1 >[2012-06-01 04:23:55.264308] I [client-handshake.c:1636:select_server_supported_programs] 66-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:23:55.264481] I [client-handshake.c:1433:client_setvolume_cbk] 66-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:23:55.264500] I [client-handshake.c:1445:client_setvolume_cbk] 66-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:23:55.276804] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 66 >[2012-06-01 04:23:55.276903] I [client-handshake.c:453:client_set_lk_version_cbk] 66-master-client-3: Server lk version = 1 >[2012-06-01 04:23:59.542531] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:00.871284] I [io-cache.c:1549:check_cache_size_ok] 67-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:00.871362] I [io-cache.c:1549:check_cache_size_ok] 67-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:00.871430] I [quota.c:3085:quota_parse_limits] 67-master-quota: no "limit-set" option provided >[2012-06-01 04:24:00.940341] I [client.c:2142:notify] 67-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:00.969935] I [client.c:2142:notify] 67-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:00.978532] I [client.c:2142:notify] 67-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:00.983891] I [client.c:2142:notify] 67-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:00.995193] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:00.995389] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:00.995707] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:00.996226] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 67-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:03.548313] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:04.292406] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:04.297857] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:04.302458] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:04.305576] I [client-handshake.c:1636:select_server_supported_programs] 67-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:04.307603] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:04.307647] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:04.307707] I [afr-common.c:3627:afr_notify] 67-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:24:04.307742] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:04.307765] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:04.307814] I [afr-common.c:3627:afr_notify] 67-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:04.307848] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:04.307863] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:04.308930] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-0: Server lk version = 1 >[2012-06-01 04:24:04.308962] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-1: Server lk version = 1 >[2012-06-01 04:24:04.308987] I [client-handshake.c:1433:client_setvolume_cbk] 67-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:04.309001] I [client-handshake.c:1445:client_setvolume_cbk] 67-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:04.329332] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 67 >[2012-06-01 04:24:04.337001] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-2: Server lk version = 1 >[2012-06-01 04:24:04.337639] I [client-handshake.c:453:client_set_lk_version_cbk] 67-master-client-3: Server lk version = 1 >[2012-06-01 04:24:09.880551] I [io-cache.c:1549:check_cache_size_ok] 68-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:09.881101] I [io-cache.c:1549:check_cache_size_ok] 68-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:09.890389] I [client.c:2142:notify] 68-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:09.911164] I [client.c:2142:notify] 68-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:09.927614] I [client.c:2142:notify] 68-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:09.934342] I [client.c:2142:notify] 68-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:09.939514] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:09.939844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:09.939962] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:09.940072] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 68-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:13.320227] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:13.320529] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:13.320559] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:13.320680] I [afr-common.c:3627:afr_notify] 68-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:13.320827] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-0: Server lk version = 1 >[2012-06-01 04:24:13.336061] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:13.336339] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:13.336368] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:13.336801] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-1: Server lk version = 1 >[2012-06-01 04:24:13.352824] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:13.353340] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:13.353371] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:13.353446] I [afr-common.c:3627:afr_notify] 68-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:13.353910] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-2: Server lk version = 1 >[2012-06-01 04:24:13.373035] I [client-handshake.c:1636:select_server_supported_programs] 68-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:13.373492] I [client-handshake.c:1433:client_setvolume_cbk] 68-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:13.373546] I [client-handshake.c:1445:client_setvolume_cbk] 68-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:13.399266] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 68 >[2012-06-01 04:24:13.399443] I [client-handshake.c:453:client_set_lk_version_cbk] 68-master-client-3: Server lk version = 1 >[2012-06-01 04:24:13.785493] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:20.264522] I [io-cache.c:1549:check_cache_size_ok] 69-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:20.264662] I [io-cache.c:1549:check_cache_size_ok] 69-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:20.264764] I [quota.c:3085:quota_parse_limits] 69-master-quota: no "limit-set" option provided >[2012-06-01 04:24:20.286678] I [client.c:2142:notify] 69-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:20.309071] I [client.c:2142:notify] 69-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:20.330002] I [client.c:2142:notify] 69-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:20.347913] I [client.c:2142:notify] 69-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:20.374683] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:20.374828] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:20.374979] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:20.377026] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 69-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:23.349216] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:23.383624] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:23.383939] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:23.383969] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:23.384044] I [afr-common.c:3627:afr_notify] 69-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:23.384179] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-0: Server lk version = 1 >[2012-06-01 04:24:23.402561] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:23.402972] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:23.403004] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:23.403419] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-1: Server lk version = 1 >[2012-06-01 04:24:23.410779] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:23.411081] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:23.411112] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:23.411188] I [afr-common.c:3627:afr_notify] 69-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:23.411329] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-2: Server lk version = 1 >[2012-06-01 04:24:23.416842] I [client-handshake.c:1636:select_server_supported_programs] 69-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:23.417404] I [client-handshake.c:1433:client_setvolume_cbk] 69-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:23.417437] I [client-handshake.c:1445:client_setvolume_cbk] 69-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:23.423810] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 69 >[2012-06-01 04:24:23.423911] I [client-handshake.c:453:client_set_lk_version_cbk] 69-master-client-3: Server lk version = 1 >[2012-06-01 04:24:24.432406] I [io-cache.c:1549:check_cache_size_ok] 70-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:24.432520] I [io-cache.c:1549:check_cache_size_ok] 70-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:24.444110] I [client.c:2142:notify] 70-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:24.453436] I [client.c:2142:notify] 70-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:24.466152] I [client.c:2142:notify] 70-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:24.475366] I [client.c:2142:notify] 70-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:24.489290] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:24.489416] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:24.489565] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:24.489661] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 70-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:28.425592] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:28.425901] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:28.425927] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:28.425991] I [afr-common.c:3627:afr_notify] 70-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:28.426111] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-0: Server lk version = 1 >[2012-06-01 04:24:28.435409] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:28.435645] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:28.435672] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:28.435732] I [afr-common.c:3627:afr_notify] 70-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:24:28.436159] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-3: Server lk version = 1 >[2012-06-01 04:24:28.444167] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:28.444430] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:28.444456] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:28.444646] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-1: Server lk version = 1 >[2012-06-01 04:24:28.452166] I [client-handshake.c:1636:select_server_supported_programs] 70-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:28.452388] I [client-handshake.c:1433:client_setvolume_cbk] 70-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:28.452413] I [client-handshake.c:1445:client_setvolume_cbk] 70-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:28.488674] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 70 >[2012-06-01 04:24:28.488762] I [client-handshake.c:453:client_set_lk_version_cbk] 70-master-client-2: Server lk version = 1 >[2012-06-01 04:24:41.690071] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:42.744452] I [io-cache.c:1549:check_cache_size_ok] 71-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:42.744563] I [io-cache.c:1549:check_cache_size_ok] 71-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:42.748731] I [quota.c:3085:quota_parse_limits] 71-master-quota: no "limit-set" option provided >[2012-06-01 04:24:42.765500] I [client.c:2142:notify] 71-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:42.774609] I [client.c:2142:notify] 71-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:42.781572] I [client.c:2142:notify] 71-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:42.786857] I [client.c:2142:notify] 71-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:42.792765] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:42.792904] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:42.793025] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:42.793142] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 71-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:44.883893] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:45.923204] I [io-cache.c:1549:check_cache_size_ok] 72-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:45.923323] I [io-cache.c:1549:check_cache_size_ok] 72-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:45.933817] I [client.c:2142:notify] 72-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:45.944548] I [client.c:2142:notify] 72-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:45.956970] I [client.c:2142:notify] 72-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:45.968641] I [client.c:2142:notify] 72-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:45.976115] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:45.976242] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:45.976360] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:45.976475] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 72-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:46.460585] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:46.467991] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:46.468156] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:46.468195] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:46.468310] I [afr-common.c:3627:afr_notify] 71-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:46.468771] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:46.468815] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:46.469231] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-0: Server lk version = 1 >[2012-06-01 04:24:46.469793] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-1: Server lk version = 1 >[2012-06-01 04:24:46.472828] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:46.473469] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:46.473509] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:46.473665] I [afr-common.c:3627:afr_notify] 71-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:46.473790] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-2: Server lk version = 1 >[2012-06-01 04:24:46.478513] I [client-handshake.c:1636:select_server_supported_programs] 71-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:46.478917] I [client-handshake.c:1433:client_setvolume_cbk] 71-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:46.478959] I [client-handshake.c:1445:client_setvolume_cbk] 71-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:46.484938] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 71 >[2012-06-01 04:24:46.485042] I [client-handshake.c:453:client_set_lk_version_cbk] 71-master-client-3: Server lk version = 1 >[2012-06-01 04:24:48.133466] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:49.292867] I [io-cache.c:1549:check_cache_size_ok] 73-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:49.292990] I [io-cache.c:1549:check_cache_size_ok] 73-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:49.293087] I [quota.c:3085:quota_parse_limits] 73-master-quota: no "limit-set" option provided >[2012-06-01 04:24:49.397896] I [client.c:2142:notify] 73-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:49.407980] I [client.c:2142:notify] 73-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:49.415854] I [client.c:2142:notify] 73-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:49.430161] I [client.c:2142:notify] 73-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:49.445195] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:49.445339] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:49.445481] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:49.445608] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 73-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:24:49.483382] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:49.483736] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:49.483769] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:49.483849] I [afr-common.c:3627:afr_notify] 72-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:49.483991] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-0: Server lk version = 1 >[2012-06-01 04:24:49.493492] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:49.493823] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:49.493854] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:49.494325] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-1: Server lk version = 1 >[2012-06-01 04:24:49.507639] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:49.508003] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:49.508035] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:49.508118] I [afr-common.c:3627:afr_notify] 72-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:49.508312] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-2: Server lk version = 1 >[2012-06-01 04:24:49.519163] I [client-handshake.c:1636:select_server_supported_programs] 72-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:49.519515] I [client-handshake.c:1433:client_setvolume_cbk] 72-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:49.519556] I [client-handshake.c:1445:client_setvolume_cbk] 72-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:49.737163] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 72 >[2012-06-01 04:24:49.739092] I [client-handshake.c:453:client_set_lk_version_cbk] 72-master-client-3: Server lk version = 1 >[2012-06-01 04:24:51.798057] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:24:52.533018] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:52.533364] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:24:52.533393] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:52.533487] I [afr-common.c:3627:afr_notify] 73-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:24:52.533701] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-0: Server lk version = 1 >[2012-06-01 04:24:52.541060] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:52.541474] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:24:52.541505] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:52.543116] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-1: Server lk version = 1 >[2012-06-01 04:24:52.553315] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:52.553603] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:24:52.553665] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:52.553740] I [afr-common.c:3627:afr_notify] 73-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:24:52.553874] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-2: Server lk version = 1 >[2012-06-01 04:24:52.561282] I [client-handshake.c:1636:select_server_supported_programs] 73-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:24:52.561944] I [client-handshake.c:1433:client_setvolume_cbk] 73-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:24:52.561976] I [client-handshake.c:1445:client_setvolume_cbk] 73-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:24:52.566802] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 73 >[2012-06-01 04:24:52.566899] I [client-handshake.c:453:client_set_lk_version_cbk] 73-master-client-3: Server lk version = 1 >[2012-06-01 04:24:57.851374] I [io-cache.c:1549:check_cache_size_ok] 74-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:24:57.851473] I [io-cache.c:1549:check_cache_size_ok] 74-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:24:57.864647] I [client.c:2142:notify] 74-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:57.879965] I [client.c:2142:notify] 74-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:57.887730] I [client.c:2142:notify] 74-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:24:57.902645] I [client.c:2142:notify] 74-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:24:57.912818] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:24:57.913022] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:24:57.913210] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:24:57.913600] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 74-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:00.171243] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:01.287018] I [io-cache.c:1549:check_cache_size_ok] 75-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:01.287095] I [io-cache.c:1549:check_cache_size_ok] 75-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:01.287157] I [quota.c:3085:quota_parse_limits] 75-master-quota: no "limit-set" option provided >[2012-06-01 04:25:01.364532] I [client.c:2142:notify] 75-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:01.375417] I [client.c:2142:notify] 75-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:01.386708] I [client.c:2142:notify] 75-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:01.401540] I [client.c:2142:notify] 75-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:01.415129] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:01.415196] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:01.415237] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:01.415324] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 75-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:01.574028] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:01.574252] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:01.574271] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:01.574328] I [afr-common.c:3627:afr_notify] 74-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:01.574443] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-0: Server lk version = 1 >[2012-06-01 04:25:01.579116] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:01.579327] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:01.579351] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:01.579536] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-1: Server lk version = 1 >[2012-06-01 04:25:01.584065] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:01.584577] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:01.584597] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:01.584656] I [afr-common.c:3627:afr_notify] 74-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:01.584684] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-2: Server lk version = 1 >[2012-06-01 04:25:01.588763] I [client-handshake.c:1636:select_server_supported_programs] 74-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:01.589031] I [client-handshake.c:1433:client_setvolume_cbk] 74-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:01.589051] I [client-handshake.c:1445:client_setvolume_cbk] 74-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:01.598620] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 74 >[2012-06-01 04:25:01.598745] I [client-handshake.c:453:client_set_lk_version_cbk] 74-master-client-3: Server lk version = 1 >[2012-06-01 04:25:03.621923] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:04.602551] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:04.602799] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:04.602824] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:04.602892] I [afr-common.c:3627:afr_notify] 75-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:04.602985] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-0: Server lk version = 1 >[2012-06-01 04:25:04.607242] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:04.607574] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:04.607593] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:04.608019] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-1: Server lk version = 1 >[2012-06-01 04:25:04.615045] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:04.615236] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:04.615257] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:04.615299] I [afr-common.c:3627:afr_notify] 75-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:04.615380] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-2: Server lk version = 1 >[2012-06-01 04:25:04.621555] I [client-handshake.c:1636:select_server_supported_programs] 75-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:04.629151] I [client-handshake.c:1433:client_setvolume_cbk] 75-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:04.629171] I [client-handshake.c:1445:client_setvolume_cbk] 75-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:04.634621] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 75 >[2012-06-01 04:25:04.634690] I [client-handshake.c:453:client_set_lk_version_cbk] 75-master-client-3: Server lk version = 1 >[2012-06-01 04:25:04.667501] I [io-cache.c:1549:check_cache_size_ok] 76-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:04.667627] I [io-cache.c:1549:check_cache_size_ok] 76-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:04.696535] I [client.c:2142:notify] 76-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:04.702678] I [client.c:2142:notify] 76-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:04.708247] I [client.c:2142:notify] 76-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:04.713981] I [client.c:2142:notify] 76-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:04.720405] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:04.720494] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:04.720571] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:04.720670] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 76-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:07.157475] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:08.269436] I [io-cache.c:1549:check_cache_size_ok] 77-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:08.269537] I [io-cache.c:1549:check_cache_size_ok] 77-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:08.269615] I [quota.c:3085:quota_parse_limits] 77-master-quota: no "limit-set" option provided >[2012-06-01 04:25:08.303204] I [client.c:2142:notify] 77-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:08.322124] I [client.c:2142:notify] 77-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:08.333204] I [client.c:2142:notify] 77-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:08.348814] I [client.c:2142:notify] 77-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:08.365536] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:08.365905] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:08.366387] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:08.366999] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 77-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:08.626832] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:08.627092] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:08.627116] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:08.627174] I [afr-common.c:3627:afr_notify] 76-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:08.627276] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-0: Server lk version = 1 >[2012-06-01 04:25:08.634345] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:08.634564] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:08.634587] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:08.635271] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-1: Server lk version = 1 >[2012-06-01 04:25:08.640701] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:08.640955] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:08.640978] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:08.641033] I [afr-common.c:3627:afr_notify] 76-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:08.641148] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-2: Server lk version = 1 >[2012-06-01 04:25:08.648018] I [client-handshake.c:1636:select_server_supported_programs] 76-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:08.649001] I [client-handshake.c:1433:client_setvolume_cbk] 76-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:08.649028] I [client-handshake.c:1445:client_setvolume_cbk] 76-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:08.657901] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 76 >[2012-06-01 04:25:08.657981] I [client-handshake.c:453:client_set_lk_version_cbk] 76-master-client-3: Server lk version = 1 >[2012-06-01 04:25:10.782022] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:11.654492] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:11.655059] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:11.655094] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:11.655175] I [afr-common.c:3627:afr_notify] 77-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:11.657282] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-0: Server lk version = 1 >[2012-06-01 04:25:11.665166] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:11.665906] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:11.665946] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:11.666246] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-1: Server lk version = 1 >[2012-06-01 04:25:11.691511] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:11.691827] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:11.691857] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:11.692004] I [afr-common.c:3627:afr_notify] 77-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:11.692149] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-2: Server lk version = 1 >[2012-06-01 04:25:11.700978] I [client-handshake.c:1636:select_server_supported_programs] 77-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:11.701430] I [client-handshake.c:1433:client_setvolume_cbk] 77-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:11.701461] I [client-handshake.c:1445:client_setvolume_cbk] 77-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:11.720711] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 77 >[2012-06-01 04:25:11.720792] I [client-handshake.c:453:client_set_lk_version_cbk] 77-master-client-3: Server lk version = 1 >[2012-06-01 04:25:21.929733] I [io-cache.c:1549:check_cache_size_ok] 78-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:21.929828] I [io-cache.c:1549:check_cache_size_ok] 78-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:21.937189] I [client.c:2142:notify] 78-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:21.941683] I [client.c:2142:notify] 78-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:21.955942] I [client.c:2142:notify] 78-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:21.959760] I [client.c:2142:notify] 78-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:21.964114] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:21.964222] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:21.964317] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:21.964408] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 78-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:25.718365] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:25.718645] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:25.718673] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:25.718740] I [afr-common.c:3627:afr_notify] 78-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:25.718859] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-0: Server lk version = 1 >[2012-06-01 04:25:25.725806] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:25.726401] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:25.726430] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:25.726796] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-1: Server lk version = 1 >[2012-06-01 04:25:25.734557] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:25.734815] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:25.734841] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:25.734937] I [afr-common.c:3627:afr_notify] 78-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:25.735059] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-2: Server lk version = 1 >[2012-06-01 04:25:25.740723] I [client-handshake.c:1636:select_server_supported_programs] 78-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:25.741678] I [client-handshake.c:1433:client_setvolume_cbk] 78-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:25.741706] I [client-handshake.c:1445:client_setvolume_cbk] 78-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:25.764943] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 78 >[2012-06-01 04:25:25.765033] I [client-handshake.c:453:client_set_lk_version_cbk] 78-master-client-3: Server lk version = 1 >[2012-06-01 04:25:34.246855] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:35.313453] I [io-cache.c:1549:check_cache_size_ok] 79-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:35.313581] I [io-cache.c:1549:check_cache_size_ok] 79-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:35.313680] I [quota.c:3085:quota_parse_limits] 79-master-quota: no "limit-set" option provided >[2012-06-01 04:25:35.321618] I [client.c:2142:notify] 79-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:35.326539] I [client.c:2142:notify] 79-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:35.331120] I [client.c:2142:notify] 79-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:35.335679] I [client.c:2142:notify] 79-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:35.341440] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:35.341551] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:35.341672] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:35.341733] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 79-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:37.433487] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:38.468750] I [io-cache.c:1549:check_cache_size_ok] 80-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:38.468852] I [io-cache.c:1549:check_cache_size_ok] 80-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:38.493901] I [client.c:2142:notify] 80-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:38.519342] I [client.c:2142:notify] 80-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:38.533058] I [client.c:2142:notify] 80-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:38.557551] I [client.c:2142:notify] 80-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:38.567232] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:38.567390] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:38.567737] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:38.567931] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 80-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:38.756409] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:38.756818] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:38.756849] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:38.757171] I [afr-common.c:3627:afr_notify] 79-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:38.757223] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-0: Server lk version = 1 >[2012-06-01 04:25:38.761815] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:38.762143] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:38.762173] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:38.762612] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-1: Server lk version = 1 >[2012-06-01 04:25:38.767910] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:38.768232] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:38.768261] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:38.768369] I [afr-common.c:3627:afr_notify] 79-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:38.768416] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-2: Server lk version = 1 >[2012-06-01 04:25:38.773137] I [client-handshake.c:1636:select_server_supported_programs] 79-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:38.773417] I [client-handshake.c:1433:client_setvolume_cbk] 79-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:38.773446] I [client-handshake.c:1445:client_setvolume_cbk] 79-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:38.804441] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 79 >[2012-06-01 04:25:38.804574] I [client-handshake.c:453:client_set_lk_version_cbk] 79-master-client-3: Server lk version = 1 >[2012-06-01 04:25:40.814788] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:41.800000] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:41.800313] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:41.800344] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:41.800421] I [afr-common.c:3627:afr_notify] 80-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:41.800561] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-0: Server lk version = 1 >[2012-06-01 04:25:41.805604] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:41.805928] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:41.805960] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:41.806423] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-1: Server lk version = 1 >[2012-06-01 04:25:41.860726] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:41.861078] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:41.861110] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:41.861188] I [afr-common.c:3627:afr_notify] 80-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:41.861328] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-2: Server lk version = 1 >[2012-06-01 04:25:41.939481] I [client-handshake.c:1636:select_server_supported_programs] 80-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:41.939816] I [client-handshake.c:1433:client_setvolume_cbk] 80-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:41.939848] I [client-handshake.c:1445:client_setvolume_cbk] 80-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:41.945393] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 80 >[2012-06-01 04:25:41.946307] I [client-handshake.c:453:client_set_lk_version_cbk] 80-master-client-3: Server lk version = 1 >[2012-06-01 04:25:42.081328] I [io-cache.c:1549:check_cache_size_ok] 81-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:42.081448] I [io-cache.c:1549:check_cache_size_ok] 81-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:42.081544] I [quota.c:3085:quota_parse_limits] 81-master-quota: no "limit-set" option provided >[2012-06-01 04:25:42.190104] I [client.c:2142:notify] 81-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:42.206973] I [client.c:2142:notify] 81-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:42.212141] I [client.c:2142:notify] 81-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:42.227800] I [client.c:2142:notify] 81-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:42.237536] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:42.238122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:42.238261] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:42.238576] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 81-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:44.493635] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:45.956320] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:45.956728] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:45.956760] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:45.956861] I [afr-common.c:3627:afr_notify] 81-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:45.956996] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-0: Server lk version = 1 >[2012-06-01 04:25:45.964984] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:45.965447] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:45.965479] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:45.965897] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-1: Server lk version = 1 >[2012-06-01 04:25:45.972997] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:45.990063] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:45.990102] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:45.990208] I [afr-common.c:3627:afr_notify] 81-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:45.990309] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-2: Server lk version = 1 >[2012-06-01 04:25:45.990410] I [client-handshake.c:1636:select_server_supported_programs] 81-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:45.990757] I [client-handshake.c:1433:client_setvolume_cbk] 81-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:45.990792] I [client-handshake.c:1445:client_setvolume_cbk] 81-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:45.998478] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 81 >[2012-06-01 04:25:45.998577] I [client-handshake.c:453:client_set_lk_version_cbk] 81-master-client-3: Server lk version = 1 >[2012-06-01 04:25:50.571620] I [io-cache.c:1549:check_cache_size_ok] 82-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:50.571731] I [io-cache.c:1549:check_cache_size_ok] 82-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:50.614660] I [client.c:2142:notify] 82-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:50.629212] I [client.c:2142:notify] 82-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:50.638335] I [client.c:2142:notify] 82-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:50.652386] I [client.c:2142:notify] 82-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:50.671479] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:50.671718] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:50.671844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:50.671997] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 82-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:52.971334] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:53.984486] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:53.994526] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:53.994553] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:53.994613] I [afr-common.c:3627:afr_notify] 82-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:53.994717] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-0: Server lk version = 1 >[2012-06-01 04:25:53.994769] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:53.994992] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:53.995022] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:53.995409] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-1: Server lk version = 1 >[2012-06-01 04:25:54.000929] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:54.001150] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:54.001172] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:54.001226] I [afr-common.c:3627:afr_notify] 82-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:54.001326] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-2: Server lk version = 1 >[2012-06-01 04:25:54.014542] I [client-handshake.c:1636:select_server_supported_programs] 82-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:54.014759] I [client-handshake.c:1433:client_setvolume_cbk] 82-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:54.014782] I [client-handshake.c:1445:client_setvolume_cbk] 82-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:54.070842] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 82 >[2012-06-01 04:25:54.070916] I [client-handshake.c:453:client_set_lk_version_cbk] 82-master-client-3: Server lk version = 1 >[2012-06-01 04:25:54.140730] I [io-cache.c:1549:check_cache_size_ok] 83-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:54.140812] I [io-cache.c:1549:check_cache_size_ok] 83-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:54.140884] I [quota.c:3085:quota_parse_limits] 83-master-quota: no "limit-set" option provided >[2012-06-01 04:25:54.172536] I [client.c:2142:notify] 83-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:54.182149] I [client.c:2142:notify] 83-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:54.188958] I [client.c:2142:notify] 83-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:54.195787] I [client.c:2142:notify] 83-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:54.203379] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:54.203825] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:54.203916] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:54.204002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 83-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:25:57.402341] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:25:58.042439] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:58.042857] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:25:58.042935] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:58.043038] I [afr-common.c:3627:afr_notify] 83-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:25:58.043221] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-0: Server lk version = 1 >[2012-06-01 04:25:58.058355] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:58.058647] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:25:58.058677] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:58.059150] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-1: Server lk version = 1 >[2012-06-01 04:25:58.083781] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:58.084257] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:25:58.084305] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:58.084425] I [afr-common.c:3627:afr_notify] 83-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:25:58.084587] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-2: Server lk version = 1 >[2012-06-01 04:25:58.112010] I [client-handshake.c:1636:select_server_supported_programs] 83-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:25:58.112438] I [client-handshake.c:1433:client_setvolume_cbk] 83-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:25:58.112500] I [client-handshake.c:1445:client_setvolume_cbk] 83-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:25:58.136785] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 83 >[2012-06-01 04:25:58.136913] I [client-handshake.c:453:client_set_lk_version_cbk] 83-master-client-3: Server lk version = 1 >[2012-06-01 04:25:58.553958] I [io-cache.c:1549:check_cache_size_ok] 84-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:25:58.554069] I [io-cache.c:1549:check_cache_size_ok] 84-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:25:58.784168] I [client.c:2142:notify] 84-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:58.803671] I [client.c:2142:notify] 84-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:58.831921] I [client.c:2142:notify] 84-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:25:58.841943] I [client.c:2142:notify] 84-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:25:58.858797] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:25:58.859312] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:25:58.859445] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:25:58.859591] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 84-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:02.132125] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:02.142556] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:02.147790] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:02.152918] I [client-handshake.c:1636:select_server_supported_programs] 84-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:02.171527] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:02.171577] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:02.171726] I [afr-common.c:3627:afr_notify] 84-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:02.171807] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:02.171838] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:02.171952] I [afr-common.c:3627:afr_notify] 84-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:02.172007] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:02.172029] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:02.195879] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-0: Server lk version = 1 >[2012-06-01 04:26:02.196019] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-2: Server lk version = 1 >[2012-06-01 04:26:02.196052] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-1: Server lk version = 1 >[2012-06-01 04:26:02.196092] I [client-handshake.c:1433:client_setvolume_cbk] 84-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:02.196114] I [client-handshake.c:1445:client_setvolume_cbk] 84-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:02.224845] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 84 >[2012-06-01 04:26:02.230254] I [client-handshake.c:453:client_set_lk_version_cbk] 84-master-client-3: Server lk version = 1 >[2012-06-01 04:26:03.397177] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:04.915560] I [io-cache.c:1549:check_cache_size_ok] 85-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:04.915706] I [io-cache.c:1549:check_cache_size_ok] 85-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:04.915826] I [quota.c:3085:quota_parse_limits] 85-master-quota: no "limit-set" option provided >[2012-06-01 04:26:04.988267] I [client.c:2142:notify] 85-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:05.019972] I [client.c:2142:notify] 85-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:05.053913] I [client.c:2142:notify] 85-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:05.106986] I [client.c:2142:notify] 85-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:05.126674] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:05.126773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:05.139510] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:05.139682] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 85-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:08.165445] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:08.173672] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:08.179292] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:08.184521] I [client-handshake.c:1636:select_server_supported_programs] 85-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:08.201965] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:08.202007] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:08.202118] I [afr-common.c:3627:afr_notify] 85-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:08.202177] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:08.202199] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:08.226573] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:08.226657] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:08.226741] I [afr-common.c:3627:afr_notify] 85-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:26:08.226795] I [client-handshake.c:1433:client_setvolume_cbk] 85-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:08.226817] I [client-handshake.c:1445:client_setvolume_cbk] 85-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:08.237035] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 85 >[2012-06-01 04:26:08.242580] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-1: Server lk version = 1 >[2012-06-01 04:26:08.242630] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-3: Server lk version = 1 >[2012-06-01 04:26:08.242663] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-2: Server lk version = 1 >[2012-06-01 04:26:08.242694] I [client-handshake.c:453:client_set_lk_version_cbk] 85-master-client-0: Server lk version = 1 >[2012-06-01 04:26:10.247056] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:12.637846] I [io-cache.c:1549:check_cache_size_ok] 86-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:12.638073] I [io-cache.c:1549:check_cache_size_ok] 86-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:12.671248] I [client.c:2142:notify] 86-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:12.686194] I [client.c:2142:notify] 86-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:12.696335] I [client.c:2142:notify] 86-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:12.717060] I [client.c:2142:notify] 86-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:12.740225] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:12.740419] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:12.740647] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:12.740810] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 86-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:16.092034] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:16.254415] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:16.254751] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:16.254781] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:16.254857] I [afr-common.c:3627:afr_notify] 86-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:16.255035] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-0: Server lk version = 1 >[2012-06-01 04:26:16.265385] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:16.265728] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:16.265758] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:16.266628] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-1: Server lk version = 1 >[2012-06-01 04:26:16.278916] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:16.279286] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:16.279315] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:16.279416] I [afr-common.c:3627:afr_notify] 86-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:16.279557] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-2: Server lk version = 1 >[2012-06-01 04:26:16.285264] I [client-handshake.c:1636:select_server_supported_programs] 86-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:16.285747] I [client-handshake.c:1433:client_setvolume_cbk] 86-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:16.285779] I [client-handshake.c:1445:client_setvolume_cbk] 86-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:16.291612] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 86 >[2012-06-01 04:26:16.291726] I [client-handshake.c:453:client_set_lk_version_cbk] 86-master-client-3: Server lk version = 1 >[2012-06-01 04:26:17.370763] I [io-cache.c:1549:check_cache_size_ok] 87-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:17.372805] I [io-cache.c:1549:check_cache_size_ok] 87-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:17.372947] I [quota.c:3085:quota_parse_limits] 87-master-quota: no "limit-set" option provided >[2012-06-01 04:26:17.400938] I [client.c:2142:notify] 87-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:17.417149] I [client.c:2142:notify] 87-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:17.432096] I [client.c:2142:notify] 87-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:17.436617] I [client.c:2142:notify] 87-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:17.464311] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:17.464474] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:17.464598] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:17.467559] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 87-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:20.425585] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:21.296042] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:21.296411] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:21.296456] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:21.296536] I [afr-common.c:3627:afr_notify] 87-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:21.296753] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-0: Server lk version = 1 >[2012-06-01 04:26:21.309093] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:21.309371] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:21.309400] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:21.309842] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-1: Server lk version = 1 >[2012-06-01 04:26:21.331229] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:21.331876] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:21.331908] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:21.331987] I [afr-common.c:3627:afr_notify] 87-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:21.332125] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-2: Server lk version = 1 >[2012-06-01 04:26:21.338741] I [client-handshake.c:1636:select_server_supported_programs] 87-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:21.339062] I [client-handshake.c:1433:client_setvolume_cbk] 87-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:21.339094] I [client-handshake.c:1445:client_setvolume_cbk] 87-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:21.701067] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 87 >[2012-06-01 04:26:21.701166] I [client-handshake.c:453:client_set_lk_version_cbk] 87-master-client-3: Server lk version = 1 >[2012-06-01 04:26:21.703605] I [io-cache.c:1549:check_cache_size_ok] 88-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:21.703721] I [io-cache.c:1549:check_cache_size_ok] 88-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:21.733231] I [client.c:2142:notify] 88-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:21.753931] I [client.c:2142:notify] 88-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:21.769114] I [client.c:2142:notify] 88-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:21.793138] I [client.c:2142:notify] 88-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:21.838410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:21.838585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:21.838717] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:21.838844] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 88-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:25.361085] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:25.361388] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:25.361422] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:25.361538] I [afr-common.c:3627:afr_notify] 88-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:25.361698] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-0: Server lk version = 1 >[2012-06-01 04:26:25.375068] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:25.375344] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:25.375368] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:25.375598] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-1: Server lk version = 1 >[2012-06-01 04:26:25.383654] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:25.383881] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:25.383902] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:25.383969] I [afr-common.c:3627:afr_notify] 88-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:25.384001] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-2: Server lk version = 1 >[2012-06-01 04:26:25.403093] I [client-handshake.c:1636:select_server_supported_programs] 88-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:25.403300] I [client-handshake.c:1433:client_setvolume_cbk] 88-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:25.403321] I [client-handshake.c:1445:client_setvolume_cbk] 88-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:25.437227] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 88 >[2012-06-01 04:26:25.458031] I [client-handshake.c:453:client_set_lk_version_cbk] 88-master-client-3: Server lk version = 1 >[2012-06-01 04:26:29.549645] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:30.891176] I [io-cache.c:1549:check_cache_size_ok] 89-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:30.891289] I [io-cache.c:1549:check_cache_size_ok] 89-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:30.891378] I [quota.c:3085:quota_parse_limits] 89-master-quota: no "limit-set" option provided >[2012-06-01 04:26:30.969162] I [client.c:2142:notify] 89-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:30.991104] I [client.c:2142:notify] 89-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:31.011658] I [client.c:2142:notify] 89-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:31.027545] I [client.c:2142:notify] 89-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:31.051119] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:31.051425] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:31.051667] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:31.051906] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 89-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:34.413141] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:34.413904] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:34.413951] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:34.414068] I [afr-common.c:3627:afr_notify] 89-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:34.414173] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-0: Server lk version = 1 >[2012-06-01 04:26:34.440177] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:34.440549] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:34.440579] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:34.441193] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-1: Server lk version = 1 >[2012-06-01 04:26:34.457680] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:34.458364] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:34.458397] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:34.458479] I [afr-common.c:3627:afr_notify] 89-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:34.458810] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-2: Server lk version = 1 >[2012-06-01 04:26:34.506053] I [client-handshake.c:1636:select_server_supported_programs] 89-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:34.506345] I [client-handshake.c:1433:client_setvolume_cbk] 89-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:34.506375] I [client-handshake.c:1445:client_setvolume_cbk] 89-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:34.511783] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 89 >[2012-06-01 04:26:34.520421] I [client-handshake.c:453:client_set_lk_version_cbk] 89-master-client-3: Server lk version = 1 >[2012-06-01 04:26:34.641960] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:40.925725] I [io-cache.c:1549:check_cache_size_ok] 90-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:40.925849] I [io-cache.c:1549:check_cache_size_ok] 90-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:40.932990] I [client.c:2142:notify] 90-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:40.959942] I [client.c:2142:notify] 90-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:40.964056] I [client.c:2142:notify] 90-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:40.968679] I [client.c:2142:notify] 90-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:40.972157] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:40.972256] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:40.972333] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:40.972409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 90-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:44.380546] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:44.519012] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:44.519259] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:44.519278] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:44.519329] I [afr-common.c:3627:afr_notify] 90-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:44.519398] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-0: Server lk version = 1 >[2012-06-01 04:26:44.532470] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:44.532655] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:44.532674] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:44.532948] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-1: Server lk version = 1 >[2012-06-01 04:26:44.562305] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:44.562509] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:44.562529] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:44.562602] I [afr-common.c:3627:afr_notify] 90-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:44.562648] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-2: Server lk version = 1 >[2012-06-01 04:26:44.572870] I [client-handshake.c:1636:select_server_supported_programs] 90-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:44.573054] I [client-handshake.c:1433:client_setvolume_cbk] 90-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:44.573072] I [client-handshake.c:1445:client_setvolume_cbk] 90-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:44.578166] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 90 >[2012-06-01 04:26:44.578245] I [client-handshake.c:453:client_set_lk_version_cbk] 90-master-client-3: Server lk version = 1 >[2012-06-01 04:26:45.631394] I [io-cache.c:1549:check_cache_size_ok] 91-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:45.631471] I [io-cache.c:1549:check_cache_size_ok] 91-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:45.631549] I [quota.c:3085:quota_parse_limits] 91-master-quota: no "limit-set" option provided >[2012-06-01 04:26:45.661095] I [client.c:2142:notify] 91-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:45.673166] I [client.c:2142:notify] 91-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:45.680452] I [client.c:2142:notify] 91-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:45.691558] I [client.c:2142:notify] 91-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:45.703035] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:45.703103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:45.703161] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:45.703216] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 91-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:48.702252] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:49.585781] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:49.586036] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:49.586057] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:49.586104] I [afr-common.c:3627:afr_notify] 91-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:49.586190] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-0: Server lk version = 1 >[2012-06-01 04:26:49.594565] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:49.594883] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:49.594907] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:49.594950] I [afr-common.c:3627:afr_notify] 91-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:26:49.595026] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-3: Server lk version = 1 >[2012-06-01 04:26:49.597705] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:49.597880] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:49.597898] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:49.598108] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-1: Server lk version = 1 >[2012-06-01 04:26:49.625959] I [client-handshake.c:1636:select_server_supported_programs] 91-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:49.626160] I [client-handshake.c:1433:client_setvolume_cbk] 91-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:49.626183] I [client-handshake.c:1445:client_setvolume_cbk] 91-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:49.662829] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 91 >[2012-06-01 04:26:49.662930] I [client-handshake.c:453:client_set_lk_version_cbk] 91-master-client-2: Server lk version = 1 >[2012-06-01 04:26:49.809794] I [io-cache.c:1549:check_cache_size_ok] 92-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:49.809896] I [io-cache.c:1549:check_cache_size_ok] 92-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:50.057759] I [client.c:2142:notify] 92-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:50.076242] I [client.c:2142:notify] 92-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:50.116977] I [client.c:2142:notify] 92-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:50.148694] I [client.c:2142:notify] 92-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:50.182765] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:50.182924] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:50.183030] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:50.185716] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 92-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:26:53.642929] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:53.643267] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:26:53.643298] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:53.643406] I [afr-common.c:3627:afr_notify] 92-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:26:53.643826] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-0: Server lk version = 1 >[2012-06-01 04:26:53.651557] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:53.651905] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:26:53.651936] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:53.652140] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-1: Server lk version = 1 >[2012-06-01 04:26:53.665424] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:53.665821] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:26:53.665866] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:53.666022] I [afr-common.c:3627:afr_notify] 92-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:26:53.666090] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-2: Server lk version = 1 >[2012-06-01 04:26:53.673156] I [client-handshake.c:1636:select_server_supported_programs] 92-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:26:53.673421] I [client-handshake.c:1433:client_setvolume_cbk] 92-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:26:53.673451] I [client-handshake.c:1445:client_setvolume_cbk] 92-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:26:53.683125] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 92 >[2012-06-01 04:26:53.683211] I [client-handshake.c:453:client_set_lk_version_cbk] 92-master-client-3: Server lk version = 1 >[2012-06-01 04:26:57.864600] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:26:59.274372] I [io-cache.c:1549:check_cache_size_ok] 93-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:26:59.274502] I [io-cache.c:1549:check_cache_size_ok] 93-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:26:59.274603] I [quota.c:3085:quota_parse_limits] 93-master-quota: no "limit-set" option provided >[2012-06-01 04:26:59.285782] I [client.c:2142:notify] 93-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:59.297088] I [client.c:2142:notify] 93-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:59.311843] I [client.c:2142:notify] 93-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:26:59.326475] I [client.c:2142:notify] 93-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:26:59.339735] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:26:59.340123] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:26:59.340251] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:26:59.340393] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 93-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:01.862625] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:02.679733] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:02.680000] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:02.680046] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:02.680131] I [afr-common.c:3627:afr_notify] 93-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:02.680169] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-0: Server lk version = 1 >[2012-06-01 04:27:02.691850] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:02.692070] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:02.692094] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:02.692366] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-1: Server lk version = 1 >[2012-06-01 04:27:02.719209] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:02.719530] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:02.719556] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:02.719642] I [afr-common.c:3627:afr_notify] 93-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:02.719879] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-2: Server lk version = 1 >[2012-06-01 04:27:02.736002] I [client-handshake.c:1636:select_server_supported_programs] 93-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:02.738175] I [client-handshake.c:1433:client_setvolume_cbk] 93-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:02.738206] I [client-handshake.c:1445:client_setvolume_cbk] 93-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:02.745542] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 93 >[2012-06-01 04:27:02.745640] I [client-handshake.c:453:client_set_lk_version_cbk] 93-master-client-3: Server lk version = 1 >[2012-06-01 04:27:02.940824] I [io-cache.c:1549:check_cache_size_ok] 94-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:02.940955] I [io-cache.c:1549:check_cache_size_ok] 94-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:03.009183] I [client.c:2142:notify] 94-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:03.019119] I [client.c:2142:notify] 94-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:03.033799] I [client.c:2142:notify] 94-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:03.046180] I [client.c:2142:notify] 94-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:03.056609] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:03.056773] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:03.056883] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:03.056957] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 94-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:05.589159] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:06.780598] I [io-cache.c:1549:check_cache_size_ok] 95-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:06.780707] I [io-cache.c:1549:check_cache_size_ok] 95-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:06.780783] I [quota.c:3085:quota_parse_limits] 95-master-quota: no "limit-set" option provided >[2012-06-01 04:27:06.816102] I [client.c:2142:notify] 95-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:06.826084] I [client.c:2142:notify] 95-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:06.840301] I [client.c:2142:notify] 95-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:06.849057] I [client.c:2142:notify] 95-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:06.860936] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:06.862956] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:06.862983] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:06.863237] I [afr-common.c:3627:afr_notify] 94-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:06.863281] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:06.863646] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:06.872982] I [client-handshake.c:1636:select_server_supported_programs] 94-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:06.873910] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-0: Server lk version = 1 >[2012-06-01 04:27:06.873956] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:06.873974] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:06.877089] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:06.877132] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:06.877226] I [afr-common.c:3627:afr_notify] 94-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:06.877270] I [client-handshake.c:1433:client_setvolume_cbk] 94-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:06.877289] I [client-handshake.c:1445:client_setvolume_cbk] 94-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:06.892184] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 94 >[2012-06-01 04:27:06.892257] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:06.892408] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-1: Server lk version = 1 >[2012-06-01 04:27:06.892458] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-2: Server lk version = 1 >[2012-06-01 04:27:06.892484] I [client-handshake.c:453:client_set_lk_version_cbk] 94-master-client-3: Server lk version = 1 >[2012-06-01 04:27:06.892529] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:06.892633] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:06.892741] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 95-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:09.309512] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:10.413225] I [io-cache.c:1549:check_cache_size_ok] 96-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:10.413332] I [io-cache.c:1549:check_cache_size_ok] 96-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:10.449930] I [client.c:2142:notify] 96-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:10.479626] I [client.c:2142:notify] 96-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:10.488424] I [client.c:2142:notify] 96-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:10.497219] I [client.c:2142:notify] 96-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:10.507284] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:10.507403] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:10.507508] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:10.507626] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 96-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:10.828378] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:10.829366] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:10.829407] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:10.829475] I [afr-common.c:3627:afr_notify] 95-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:10.829854] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-0: Server lk version = 1 >[2012-06-01 04:27:10.850480] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:10.851392] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:10.851421] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:10.851797] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-1: Server lk version = 1 >[2012-06-01 04:27:10.879712] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:10.880192] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:10.880219] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:10.880286] I [afr-common.c:3627:afr_notify] 95-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:10.880466] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-2: Server lk version = 1 >[2012-06-01 04:27:10.898024] I [client-handshake.c:1636:select_server_supported_programs] 95-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:10.898283] I [client-handshake.c:1433:client_setvolume_cbk] 95-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:10.898309] I [client-handshake.c:1445:client_setvolume_cbk] 95-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:10.908658] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 95 >[2012-06-01 04:27:10.908743] I [client-handshake.c:453:client_set_lk_version_cbk] 95-master-client-3: Server lk version = 1 >[2012-06-01 04:27:13.064200] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:13.911200] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:13.911537] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:13.911555] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:13.911604] I [afr-common.c:3627:afr_notify] 96-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:13.911678] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-0: Server lk version = 1 >[2012-06-01 04:27:13.926462] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:13.926756] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:13.926776] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:13.927085] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-1: Server lk version = 1 >[2012-06-01 04:27:13.934593] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:13.934830] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:13.934850] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:13.934962] I [afr-common.c:3627:afr_notify] 96-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:13.934992] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-2: Server lk version = 1 >[2012-06-01 04:27:13.947922] I [client-handshake.c:1636:select_server_supported_programs] 96-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:13.948841] I [client-handshake.c:1433:client_setvolume_cbk] 96-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:13.948861] I [client-handshake.c:1445:client_setvolume_cbk] 96-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:13.978392] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 96 >[2012-06-01 04:27:13.978473] I [client-handshake.c:453:client_set_lk_version_cbk] 96-master-client-3: Server lk version = 1 >[2012-06-01 04:27:14.446814] I [io-cache.c:1549:check_cache_size_ok] 97-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:14.446917] I [io-cache.c:1549:check_cache_size_ok] 97-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:14.446983] I [quota.c:3085:quota_parse_limits] 97-master-quota: no "limit-set" option provided >[2012-06-01 04:27:14.519172] I [client.c:2142:notify] 97-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:14.553640] I [client.c:2142:notify] 97-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:14.622154] I [client.c:2142:notify] 97-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:14.642572] I [client.c:2142:notify] 97-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:14.661998] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:14.662102] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:14.662288] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:14.662372] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 97-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:17.970341] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:17.984057] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:17.984096] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:17.984179] I [afr-common.c:3627:afr_notify] 97-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:17.984645] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-0: Server lk version = 1 >[2012-06-01 04:27:17.984840] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:17.985211] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:17.985241] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:17.995755] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-1: Server lk version = 1 >[2012-06-01 04:27:18.005393] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:18.005911] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:18.005946] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:18.006031] I [afr-common.c:3627:afr_notify] 97-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:18.006111] I [client-handshake.c:1636:select_server_supported_programs] 97-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:18.006466] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-2: Server lk version = 1 >[2012-06-01 04:27:18.006517] I [client-handshake.c:1433:client_setvolume_cbk] 97-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:18.006539] I [client-handshake.c:1445:client_setvolume_cbk] 97-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:18.023805] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 97 >[2012-06-01 04:27:18.023966] I [client-handshake.c:453:client_set_lk_version_cbk] 97-master-client-3: Server lk version = 1 >[2012-06-01 04:27:28.961159] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:30.153852] I [io-cache.c:1549:check_cache_size_ok] 98-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:30.153947] I [io-cache.c:1549:check_cache_size_ok] 98-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:30.165959] I [client.c:2142:notify] 98-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:30.182862] I [client.c:2142:notify] 98-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:30.196003] I [client.c:2142:notify] 98-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:30.208495] I [client.c:2142:notify] 98-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:30.220031] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:30.220146] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:30.220246] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:30.220343] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 98-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:33.209789] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:34.001661] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:34.002255] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:34.002275] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:34.002321] I [afr-common.c:3627:afr_notify] 98-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:34.002411] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-0: Server lk version = 1 >[2012-06-01 04:27:34.004908] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:34.005089] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:34.005112] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:34.005340] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-1: Server lk version = 1 >[2012-06-01 04:27:34.013083] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:34.013255] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:34.013279] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:34.013325] I [afr-common.c:3627:afr_notify] 98-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:34.013407] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-2: Server lk version = 1 >[2012-06-01 04:27:34.026577] I [client-handshake.c:1636:select_server_supported_programs] 98-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:34.026806] I [client-handshake.c:1433:client_setvolume_cbk] 98-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:34.026830] I [client-handshake.c:1445:client_setvolume_cbk] 98-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:34.033315] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 98 >[2012-06-01 04:27:34.033387] I [client-handshake.c:453:client_set_lk_version_cbk] 98-master-client-3: Server lk version = 1 >[2012-06-01 04:27:34.478660] I [io-cache.c:1549:check_cache_size_ok] 99-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:34.478775] I [io-cache.c:1549:check_cache_size_ok] 99-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:34.478867] I [quota.c:3085:quota_parse_limits] 99-master-quota: no "limit-set" option provided >[2012-06-01 04:27:34.499127] I [client.c:2142:notify] 99-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:34.525230] I [client.c:2142:notify] 99-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:34.541664] I [client.c:2142:notify] 99-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:34.562426] I [client.c:2142:notify] 99-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:34.596011] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:34.596192] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:34.596658] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:34.596800] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 99-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:36.930062] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:38.043900] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:38.044134] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:38.044155] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:38.044221] I [afr-common.c:3627:afr_notify] 99-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:38.044249] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-0: Server lk version = 1 >[2012-06-01 04:27:38.047006] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:38.047194] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:38.047213] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:38.047433] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-1: Server lk version = 1 >[2012-06-01 04:27:38.051191] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:38.051371] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:38.051390] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:38.051440] I [afr-common.c:3627:afr_notify] 99-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:38.051522] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-2: Server lk version = 1 >[2012-06-01 04:27:38.054561] I [client-handshake.c:1636:select_server_supported_programs] 99-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:38.071021] I [client-handshake.c:1433:client_setvolume_cbk] 99-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:38.071046] I [client-handshake.c:1445:client_setvolume_cbk] 99-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:38.075225] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 99 >[2012-06-01 04:27:38.075285] I [client-handshake.c:453:client_set_lk_version_cbk] 99-master-client-3: Server lk version = 1 >[2012-06-01 04:27:43.016941] I [io-cache.c:1549:check_cache_size_ok] 100-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:43.017106] I [io-cache.c:1549:check_cache_size_ok] 100-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:43.117862] I [client.c:2142:notify] 100-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:43.129218] I [client.c:2142:notify] 100-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:43.145764] I [client.c:2142:notify] 100-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:43.162024] I [client.c:2142:notify] 100-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:43.175145] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:43.175287] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:43.175410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:43.175692] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 100-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:45.466436] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:46.624182] I [io-cache.c:1549:check_cache_size_ok] 101-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:46.624277] I [io-cache.c:1549:check_cache_size_ok] 101-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:46.624356] I [quota.c:3085:quota_parse_limits] 101-master-quota: no "limit-set" option provided >[2012-06-01 04:27:46.648730] I [client.c:2142:notify] 101-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:46.659159] I [client.c:2142:notify] 101-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:46.668952] I [client.c:2142:notify] 101-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:46.672486] I [client.c:2142:notify] 101-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:46.684522] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:46.684656] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:46.684763] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:46.685122] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 101-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:47.066911] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:47.068446] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:47.068498] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:47.068569] I [afr-common.c:3627:afr_notify] 100-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:47.068728] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-0: Server lk version = 1 >[2012-06-01 04:27:47.076853] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:47.077094] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:47.077120] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:47.077499] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-1: Server lk version = 1 >[2012-06-01 04:27:47.086244] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:47.086525] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:47.086549] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:47.086639] I [afr-common.c:3627:afr_notify] 100-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:47.086796] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-2: Server lk version = 1 >[2012-06-01 04:27:47.092551] I [client-handshake.c:1636:select_server_supported_programs] 100-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:47.093246] I [client-handshake.c:1433:client_setvolume_cbk] 100-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:47.093279] I [client-handshake.c:1445:client_setvolume_cbk] 100-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:47.104061] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 100 >[2012-06-01 04:27:47.104444] I [client-handshake.c:453:client_set_lk_version_cbk] 100-master-client-3: Server lk version = 1 >[2012-06-01 04:27:50.098179] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:50.098486] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:50.098516] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:50.098607] I [afr-common.c:3627:afr_notify] 101-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:50.098745] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-0: Server lk version = 1 >[2012-06-01 04:27:50.104556] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:50.104901] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:50.104932] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:50.105277] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-1: Server lk version = 1 >[2012-06-01 04:27:50.113244] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:50.113540] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:50.113585] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:50.113693] I [afr-common.c:3627:afr_notify] 101-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:50.113836] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-2: Server lk version = 1 >[2012-06-01 04:27:50.124170] I [client-handshake.c:1636:select_server_supported_programs] 101-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:50.125096] I [client-handshake.c:1433:client_setvolume_cbk] 101-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:50.125132] I [client-handshake.c:1445:client_setvolume_cbk] 101-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:50.130228] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 101 >[2012-06-01 04:27:50.130318] I [client-handshake.c:453:client_set_lk_version_cbk] 101-master-client-3: Server lk version = 1 >[2012-06-01 04:27:54.099641] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:55.156475] I [io-cache.c:1549:check_cache_size_ok] 102-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:55.156907] I [io-cache.c:1549:check_cache_size_ok] 102-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:55.185537] I [client.c:2142:notify] 102-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:55.204068] I [client.c:2142:notify] 102-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:55.211008] I [client.c:2142:notify] 102-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:55.217334] I [client.c:2142:notify] 102-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:55.241210] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:55.241362] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:55.242970] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:55.243112] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 102-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:57.540847] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:27:58.712185] I [io-cache.c:1549:check_cache_size_ok] 103-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:27:58.712280] I [io-cache.c:1549:check_cache_size_ok] 103-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:27:58.712352] I [quota.c:3085:quota_parse_limits] 103-master-quota: no "limit-set" option provided >[2012-06-01 04:27:58.775970] I [client.c:2142:notify] 103-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:58.793424] I [client.c:2142:notify] 103-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:58.806933] I [client.c:2142:notify] 103-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:27:58.824123] I [client.c:2142:notify] 103-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:27:58.850289] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:27:58.850402] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:27:58.850515] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:27:58.850643] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 103-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:27:59.129890] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:59.131154] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:27:59.131178] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:59.131234] I [afr-common.c:3627:afr_notify] 102-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:27:59.131330] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-0: Server lk version = 1 >[2012-06-01 04:27:59.138167] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:59.138380] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:27:59.138403] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:59.138705] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-1: Server lk version = 1 >[2012-06-01 04:27:59.143261] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:59.143473] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:27:59.143515] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:59.143575] I [afr-common.c:3627:afr_notify] 102-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:27:59.143700] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-2: Server lk version = 1 >[2012-06-01 04:27:59.147933] I [client-handshake.c:1636:select_server_supported_programs] 102-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:27:59.148168] I [client-handshake.c:1433:client_setvolume_cbk] 102-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:27:59.148191] I [client-handshake.c:1445:client_setvolume_cbk] 102-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:27:59.155607] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 102 >[2012-06-01 04:27:59.155773] I [client-handshake.c:453:client_set_lk_version_cbk] 102-master-client-3: Server lk version = 1 >[2012-06-01 04:28:01.245889] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:02.160252] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:02.160547] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:02.160569] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:02.160620] I [afr-common.c:3627:afr_notify] 103-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:02.160805] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-0: Server lk version = 1 >[2012-06-01 04:28:02.179281] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:02.179508] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:02.179527] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:02.179660] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-1: Server lk version = 1 >[2012-06-01 04:28:02.195837] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:02.196108] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:02.196128] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:02.196176] I [afr-common.c:3627:afr_notify] 103-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:02.196263] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-2: Server lk version = 1 >[2012-06-01 04:28:02.199509] I [client-handshake.c:1636:select_server_supported_programs] 103-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:02.199710] I [client-handshake.c:1433:client_setvolume_cbk] 103-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:02.199730] I [client-handshake.c:1445:client_setvolume_cbk] 103-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:02.218380] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 103 >[2012-06-01 04:28:02.218459] I [client-handshake.c:453:client_set_lk_version_cbk] 103-master-client-3: Server lk version = 1 >[2012-06-01 04:28:02.390672] I [io-cache.c:1549:check_cache_size_ok] 104-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:02.390755] I [io-cache.c:1549:check_cache_size_ok] 104-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:02.398676] I [client.c:2142:notify] 104-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:02.411779] I [client.c:2142:notify] 104-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:02.417752] I [client.c:2142:notify] 104-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:02.425577] I [client.c:2142:notify] 104-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:02.435330] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:02.435422] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:02.435504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:02.435593] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 104-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:04.655458] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:05.892336] I [io-cache.c:1549:check_cache_size_ok] 105-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:05.892472] I [io-cache.c:1549:check_cache_size_ok] 105-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:05.892567] I [quota.c:3085:quota_parse_limits] 105-master-quota: no "limit-set" option provided >[2012-06-01 04:28:05.902787] I [client.c:2142:notify] 105-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:05.923724] I [client.c:2142:notify] 105-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:05.938732] I [client.c:2142:notify] 105-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:05.945863] I [client.c:2142:notify] 105-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:05.956878] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:05.957030] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:05.957152] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:05.961959] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 105-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:06.224693] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:06.225025] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:06.225054] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:06.225131] I [afr-common.c:3627:afr_notify] 104-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:06.225262] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-0: Server lk version = 1 >[2012-06-01 04:28:06.242495] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:06.242817] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:06.242847] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:06.243144] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-1: Server lk version = 1 >[2012-06-01 04:28:06.252151] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:06.252441] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:06.252484] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:06.252562] I [afr-common.c:3627:afr_notify] 104-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:06.253033] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-2: Server lk version = 1 >[2012-06-01 04:28:06.267060] I [client-handshake.c:1636:select_server_supported_programs] 104-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:06.267336] I [client-handshake.c:1433:client_setvolume_cbk] 104-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:06.267365] I [client-handshake.c:1445:client_setvolume_cbk] 104-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:06.273478] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 104 >[2012-06-01 04:28:06.273567] I [client-handshake.c:453:client_set_lk_version_cbk] 104-master-client-3: Server lk version = 1 >[2012-06-01 04:28:08.296843] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:09.280315] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:09.281167] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:09.281811] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:09.281930] I [afr-common.c:3627:afr_notify] 105-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:09.282267] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-0: Server lk version = 1 >[2012-06-01 04:28:09.291784] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:09.292978] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:09.293009] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:09.293610] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-1: Server lk version = 1 >[2012-06-01 04:28:09.300516] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:09.300812] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:09.300845] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:09.300945] I [afr-common.c:3627:afr_notify] 105-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:09.301095] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-2: Server lk version = 1 >[2012-06-01 04:28:09.306946] I [client-handshake.c:1636:select_server_supported_programs] 105-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:09.307224] I [client-handshake.c:1433:client_setvolume_cbk] 105-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:09.307259] I [client-handshake.c:1445:client_setvolume_cbk] 105-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:09.312556] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 105 >[2012-06-01 04:28:09.312741] I [client-handshake.c:453:client_set_lk_version_cbk] 105-master-client-3: Server lk version = 1 >[2012-06-01 04:28:09.378667] I [io-cache.c:1549:check_cache_size_ok] 106-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:09.378794] I [io-cache.c:1549:check_cache_size_ok] 106-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:09.433951] I [client.c:2142:notify] 106-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:09.466883] I [client.c:2142:notify] 106-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:09.480141] I [client.c:2142:notify] 106-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:09.487812] I [client.c:2142:notify] 106-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:09.509123] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:09.513099] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:09.513942] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:09.514063] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 106-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:11.855327] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:13.083841] I [io-cache.c:1549:check_cache_size_ok] 107-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:13.083937] I [io-cache.c:1549:check_cache_size_ok] 107-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:13.084001] I [quota.c:3085:quota_parse_limits] 107-master-quota: no "limit-set" option provided >[2012-06-01 04:28:13.110796] I [client.c:2142:notify] 107-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:13.114013] I [client.c:2142:notify] 107-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:13.125990] I [client.c:2142:notify] 107-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:13.130448] I [client.c:2142:notify] 107-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:13.136989] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:13.137079] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:13.137159] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:13.137311] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 107-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:13.319388] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:13.319609] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:13.319644] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:13.319696] I [afr-common.c:3627:afr_notify] 106-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:13.320037] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-0: Server lk version = 1 >[2012-06-01 04:28:13.328978] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:13.329168] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:13.329187] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:13.329366] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-1: Server lk version = 1 >[2012-06-01 04:28:13.339992] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:13.340556] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:13.340580] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:13.340644] I [afr-common.c:3627:afr_notify] 106-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:13.340811] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-2: Server lk version = 1 >[2012-06-01 04:28:13.347885] I [client-handshake.c:1636:select_server_supported_programs] 106-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:13.348171] I [client-handshake.c:1433:client_setvolume_cbk] 106-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:13.348190] I [client-handshake.c:1445:client_setvolume_cbk] 106-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:13.352233] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 106 >[2012-06-01 04:28:13.352303] I [client-handshake.c:453:client_set_lk_version_cbk] 106-master-client-3: Server lk version = 1 >[2012-06-01 04:28:15.386738] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:16.353252] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:16.353578] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:16.353605] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:16.353673] I [afr-common.c:3627:afr_notify] 107-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:16.353796] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-0: Server lk version = 1 >[2012-06-01 04:28:16.366437] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:16.383532] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:16.383563] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:16.383766] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-1: Server lk version = 1 >[2012-06-01 04:28:16.384043] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:16.384139] I [client-handshake.c:1636:select_server_supported_programs] 107-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:16.384404] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:16.384431] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:16.384511] I [afr-common.c:3627:afr_notify] 107-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:28:16.384641] I [client-handshake.c:1433:client_setvolume_cbk] 107-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:16.384668] I [client-handshake.c:1445:client_setvolume_cbk] 107-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:16.392567] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 107 >[2012-06-01 04:28:16.392655] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-3: Server lk version = 1 >[2012-06-01 04:28:16.392703] I [client-handshake.c:453:client_set_lk_version_cbk] 107-master-client-2: Server lk version = 1 >[2012-06-01 04:28:16.459605] I [io-cache.c:1549:check_cache_size_ok] 108-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:16.459706] I [io-cache.c:1549:check_cache_size_ok] 108-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:16.681414] I [client.c:2142:notify] 108-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:16.712235] I [client.c:2142:notify] 108-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:16.721451] I [client.c:2142:notify] 108-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:16.741437] I [client.c:2142:notify] 108-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:16.752036] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:16.752159] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:16.752268] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:16.752601] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 108-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:20.383510] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:20.384184] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:20.384205] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:20.384253] I [afr-common.c:3627:afr_notify] 108-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:20.384336] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-0: Server lk version = 1 >[2012-06-01 04:28:20.390589] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:20.390833] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:20.390854] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:20.391069] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-1: Server lk version = 1 >[2012-06-01 04:28:20.394551] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:20.395365] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:20.395386] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:20.395436] I [afr-common.c:3627:afr_notify] 108-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:20.395571] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-2: Server lk version = 1 >[2012-06-01 04:28:20.404458] I [client-handshake.c:1636:select_server_supported_programs] 108-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:20.404670] I [client-handshake.c:1433:client_setvolume_cbk] 108-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:20.404691] I [client-handshake.c:1445:client_setvolume_cbk] 108-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:20.418471] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 108 >[2012-06-01 04:28:20.418571] I [client-handshake.c:453:client_set_lk_version_cbk] 108-master-client-3: Server lk version = 1 >[2012-06-01 04:28:28.989496] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:30.322685] I [io-cache.c:1549:check_cache_size_ok] 109-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:30.322797] I [io-cache.c:1549:check_cache_size_ok] 109-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:30.322888] I [quota.c:3085:quota_parse_limits] 109-master-quota: no "limit-set" option provided >[2012-06-01 04:28:30.364900] I [client.c:2142:notify] 109-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:30.379772] I [client.c:2142:notify] 109-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:30.389501] I [client.c:2142:notify] 109-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:30.398231] I [client.c:2142:notify] 109-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:30.420305] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:30.420406] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:30.420539] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:30.420622] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 109-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:33.537234] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:34.421675] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:34.422006] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:34.422036] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:34.422112] I [afr-common.c:3627:afr_notify] 109-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:34.422251] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-0: Server lk version = 1 >[2012-06-01 04:28:34.432943] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:34.433229] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:34.433258] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:34.433331] I [afr-common.c:3627:afr_notify] 109-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:34.433465] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-2: Server lk version = 1 >[2012-06-01 04:28:34.438330] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:34.438635] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:34.438668] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:34.439067] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-3: Server lk version = 1 >[2012-06-01 04:28:34.439777] I [client-handshake.c:1636:select_server_supported_programs] 109-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:34.440109] I [client-handshake.c:1433:client_setvolume_cbk] 109-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:34.440140] I [client-handshake.c:1445:client_setvolume_cbk] 109-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:34.456686] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 109 >[2012-06-01 04:28:34.456764] I [client-handshake.c:453:client_set_lk_version_cbk] 109-master-client-1: Server lk version = 1 >[2012-06-01 04:28:34.631541] I [io-cache.c:1549:check_cache_size_ok] 110-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:34.633947] I [io-cache.c:1549:check_cache_size_ok] 110-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:34.656885] I [client.c:2142:notify] 110-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:34.675911] I [client.c:2142:notify] 110-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:34.684602] I [client.c:2142:notify] 110-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:34.700030] I [client.c:2142:notify] 110-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:34.721929] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:34.722133] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:34.722218] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:34.722321] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 110-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:37.016461] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:38.137220] I [io-cache.c:1549:check_cache_size_ok] 111-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:38.137337] I [io-cache.c:1549:check_cache_size_ok] 111-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:38.137442] I [quota.c:3085:quota_parse_limits] 111-master-quota: no "limit-set" option provided >[2012-06-01 04:28:38.178533] I [client.c:2142:notify] 111-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:38.190471] I [client.c:2142:notify] 111-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:38.202567] I [client.c:2142:notify] 111-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:38.211659] I [client.c:2142:notify] 111-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:38.226278] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:38.226443] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:38.226579] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:38.226709] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 111-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:38.447466] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:38.447814] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:38.447844] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:38.447959] I [afr-common.c:3627:afr_notify] 110-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:38.448108] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-0: Server lk version = 1 >[2012-06-01 04:28:38.455268] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:38.455572] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:38.455604] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:38.455985] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-1: Server lk version = 1 >[2012-06-01 04:28:38.464073] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:38.464362] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:38.464403] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:38.464497] I [afr-common.c:3627:afr_notify] 110-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:38.464640] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-2: Server lk version = 1 >[2012-06-01 04:28:38.471750] I [client-handshake.c:1636:select_server_supported_programs] 110-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:38.472189] I [client-handshake.c:1433:client_setvolume_cbk] 110-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:38.472223] I [client-handshake.c:1445:client_setvolume_cbk] 110-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:38.481575] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 110 >[2012-06-01 04:28:38.481666] I [client-handshake.c:453:client_set_lk_version_cbk] 110-master-client-3: Server lk version = 1 >[2012-06-01 04:28:41.480297] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:41.480851] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:41.480914] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:41.480998] I [afr-common.c:3627:afr_notify] 111-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:41.481828] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-0: Server lk version = 1 >[2012-06-01 04:28:41.489947] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:41.490271] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:41.490303] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:41.490509] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-1: Server lk version = 1 >[2012-06-01 04:28:41.499368] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:41.499769] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:41.499802] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:41.499920] I [afr-common.c:3627:afr_notify] 111-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:41.500157] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-2: Server lk version = 1 >[2012-06-01 04:28:41.502746] I [client-handshake.c:1636:select_server_supported_programs] 111-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:41.503694] I [client-handshake.c:1433:client_setvolume_cbk] 111-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:41.503729] I [client-handshake.c:1445:client_setvolume_cbk] 111-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:41.516756] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 111 >[2012-06-01 04:28:41.516857] I [client-handshake.c:453:client_set_lk_version_cbk] 111-master-client-3: Server lk version = 1 >[2012-06-01 04:28:45.444759] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:46.512374] I [io-cache.c:1549:check_cache_size_ok] 112-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:46.512478] I [io-cache.c:1549:check_cache_size_ok] 112-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:46.566693] I [client.c:2142:notify] 112-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:46.576323] I [client.c:2142:notify] 112-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:46.587778] I [client.c:2142:notify] 112-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:46.594735] I [client.c:2142:notify] 112-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:46.621525] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:46.622128] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:46.622677] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:46.622949] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 112-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:48.895706] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:50.012830] I [io-cache.c:1549:check_cache_size_ok] 113-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:50.012973] I [io-cache.c:1549:check_cache_size_ok] 113-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:50.013047] I [quota.c:3085:quota_parse_limits] 113-master-quota: no "limit-set" option provided >[2012-06-01 04:28:50.089850] I [client.c:2142:notify] 113-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:50.105617] I [client.c:2142:notify] 113-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:50.123799] I [client.c:2142:notify] 113-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:50.127304] I [client.c:2142:notify] 113-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:50.147179] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:50.147296] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:50.147464] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:50.147772] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 113-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:50.513426] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:50.513812] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:50.513855] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:50.513979] I [afr-common.c:3627:afr_notify] 112-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:50.514027] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-0: Server lk version = 1 >[2012-06-01 04:28:50.522681] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:50.523005] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:50.523037] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:50.523264] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-1: Server lk version = 1 >[2012-06-01 04:28:50.531152] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:50.531470] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:50.531512] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:50.531590] I [afr-common.c:3627:afr_notify] 112-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:50.531665] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-2: Server lk version = 1 >[2012-06-01 04:28:50.537423] I [client-handshake.c:1636:select_server_supported_programs] 112-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:50.537882] I [client-handshake.c:1433:client_setvolume_cbk] 112-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:50.537921] I [client-handshake.c:1445:client_setvolume_cbk] 112-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:50.550387] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 112 >[2012-06-01 04:28:50.550561] I [client-handshake.c:453:client_set_lk_version_cbk] 112-master-client-3: Server lk version = 1 >[2012-06-01 04:28:52.283472] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:28:53.349563] I [io-cache.c:1549:check_cache_size_ok] 114-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:28:53.362816] I [io-cache.c:1549:check_cache_size_ok] 114-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:28:53.478003] I [client.c:2142:notify] 114-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:53.504183] I [client.c:2142:notify] 114-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:53.517485] I [client.c:2142:notify] 114-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:28:53.533113] I [client.c:2142:notify] 114-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:28:53.555211] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:28:53.555428] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:28:53.555637] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:28:53.555771] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 114-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:28:53.563447] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:53.563754] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:53.563786] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:53.563898] I [afr-common.c:3627:afr_notify] 113-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:53.564048] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-0: Server lk version = 1 >[2012-06-01 04:28:53.577959] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:53.578446] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:53.578501] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:53.579036] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-1: Server lk version = 1 >[2012-06-01 04:28:53.587517] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:53.614384] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:53.614457] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:53.614706] I [afr-common.c:3627:afr_notify] 113-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:53.621557] I [client-handshake.c:1636:select_server_supported_programs] 113-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:53.622073] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-2: Server lk version = 1 >[2012-06-01 04:28:53.624739] I [client-handshake.c:1433:client_setvolume_cbk] 113-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:53.624772] I [client-handshake.c:1445:client_setvolume_cbk] 113-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:53.710682] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 113 >[2012-06-01 04:28:53.710844] I [client-handshake.c:453:client_set_lk_version_cbk] 113-master-client-3: Server lk version = 1 >[2012-06-01 04:28:56.608388] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:56.609205] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:28:56.609237] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:56.609315] I [afr-common.c:3627:afr_notify] 114-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:28:56.609464] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-0: Server lk version = 1 >[2012-06-01 04:28:56.618756] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:56.619294] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:28:56.619329] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:56.619566] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-1: Server lk version = 1 >[2012-06-01 04:28:56.627512] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:56.629564] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:28:56.629598] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:56.629727] I [afr-common.c:3627:afr_notify] 114-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:28:56.629789] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-2: Server lk version = 1 >[2012-06-01 04:28:56.639756] I [client-handshake.c:1636:select_server_supported_programs] 114-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:28:56.640048] I [client-handshake.c:1433:client_setvolume_cbk] 114-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:28:56.640078] I [client-handshake.c:1445:client_setvolume_cbk] 114-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:28:56.701840] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 114 >[2012-06-01 04:28:56.702086] I [client-handshake.c:453:client_set_lk_version_cbk] 114-master-client-3: Server lk version = 1 >[2012-06-01 04:28:57.783821] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:00.248435] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 114-master-replicate-0: added root inode >[2012-06-01 04:29:00.249607] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 114-master-replicate-1: added root inode >[2012-06-01 04:29:00.305831] I [client.c:2151:notify] 58-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:00.311833] I [client.c:2151:notify] 58-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:00.311933] I [client.c:2151:notify] 58-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:00.311988] I [client.c:2151:notify] 58-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:00.311940] I [client.c:2090:client_rpc_notify] 58-master-client-0: disconnected >[2012-06-01 04:29:00.326726] I [client.c:2090:client_rpc_notify] 58-master-client-1: disconnected >[2012-06-01 04:29:00.326753] E [afr-common.c:3664:afr_notify] 58-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:29:00.353944] I [client.c:2090:client_rpc_notify] 58-master-client-2: disconnected >[2012-06-01 04:29:00.354023] I [client.c:2090:client_rpc_notify] 58-master-client-3: disconnected >[2012-06-01 04:29:00.354048] E [afr-common.c:3664:afr_notify] 58-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:29:00.572980] I [io-cache.c:1549:check_cache_size_ok] 115-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:00.573106] I [io-cache.c:1549:check_cache_size_ok] 115-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:00.573239] I [quota.c:3085:quota_parse_limits] 115-master-quota: no "limit-set" option provided >[2012-06-01 04:29:00.589808] I [client.c:2142:notify] 115-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:00.596482] I [client.c:2142:notify] 115-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:00.615350] I [client.c:2142:notify] 115-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:00.626706] I [client.c:2142:notify] 115-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:00.661903] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:00.662028] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:00.662105] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:00.662205] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 115-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:03.669980] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:03.680114] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:03.684974] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:03.690470] I [client-handshake.c:1636:select_server_supported_programs] 115-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:03.720654] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:03.720698] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:03.720818] I [afr-common.c:3627:afr_notify] 115-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:03.720905] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:03.720932] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:03.752618] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:03.752662] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:03.752758] I [afr-common.c:3627:afr_notify] 115-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:03.752810] I [client-handshake.c:1433:client_setvolume_cbk] 115-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:03.752832] I [client-handshake.c:1445:client_setvolume_cbk] 115-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:03.772699] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 115 >[2012-06-01 04:29:03.783977] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-2: Server lk version = 1 >[2012-06-01 04:29:03.784021] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-3: Server lk version = 1 >[2012-06-01 04:29:03.784054] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-0: Server lk version = 1 >[2012-06-01 04:29:03.784418] I [client-handshake.c:453:client_set_lk_version_cbk] 115-master-client-1: Server lk version = 1 >[2012-06-01 04:29:10.905512] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:12.678376] I [io-cache.c:1549:check_cache_size_ok] 116-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:12.678498] I [io-cache.c:1549:check_cache_size_ok] 116-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:12.693815] I [client.c:2142:notify] 116-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:12.705993] I [client.c:2142:notify] 116-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:12.715565] I [client.c:2142:notify] 116-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:12.727923] I [client.c:2142:notify] 116-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:12.738339] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:12.738497] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:12.738622] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:12.738764] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 116-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:16.710905] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:16.711275] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:16.711307] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:16.711390] I [afr-common.c:3627:afr_notify] 116-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:16.711542] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-0: Server lk version = 1 >[2012-06-01 04:29:16.743062] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:16.743383] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:16.743415] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:16.743806] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-1: Server lk version = 1 >[2012-06-01 04:29:16.759581] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:16.760015] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:16.760049] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:16.760156] I [afr-common.c:3627:afr_notify] 116-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:16.760307] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-2: Server lk version = 1 >[2012-06-01 04:29:16.792155] I [client-handshake.c:1636:select_server_supported_programs] 116-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:16.792541] I [client-handshake.c:1433:client_setvolume_cbk] 116-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:16.792575] I [client-handshake.c:1445:client_setvolume_cbk] 116-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:16.867340] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 116 >[2012-06-01 04:29:16.867458] I [client-handshake.c:453:client_set_lk_version_cbk] 116-master-client-3: Server lk version = 1 >[2012-06-01 04:29:25.842237] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:32.030235] I [io-cache.c:1549:check_cache_size_ok] 117-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:32.030334] I [io-cache.c:1549:check_cache_size_ok] 117-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:32.030425] I [quota.c:3085:quota_parse_limits] 117-master-quota: no "limit-set" option provided >[2012-06-01 04:29:32.104130] I [client.c:2142:notify] 117-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:32.117570] I [client.c:2142:notify] 117-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:32.136826] I [client.c:2142:notify] 117-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:32.154292] I [client.c:2142:notify] 117-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:32.181353] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:32.181557] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:32.181700] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:32.181809] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 117-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:34.488179] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:35.552312] I [io-cache.c:1549:check_cache_size_ok] 118-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:35.552419] I [io-cache.c:1549:check_cache_size_ok] 118-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:35.622708] I [client.c:2142:notify] 118-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:35.643242] I [client.c:2142:notify] 118-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:35.664277] I [client.c:2142:notify] 118-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:35.676976] I [client.c:2142:notify] 118-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:35.691385] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:35.691483] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:35.691580] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:35.691691] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 118-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:35.806235] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:35.806518] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:35.806547] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:35.806647] I [afr-common.c:3627:afr_notify] 117-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:35.806774] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-0: Server lk version = 1 >[2012-06-01 04:29:35.848539] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:35.848858] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:35.848885] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:35.849321] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-1: Server lk version = 1 >[2012-06-01 04:29:35.892857] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:35.893123] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:35.893151] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:35.893224] I [afr-common.c:3627:afr_notify] 117-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:35.893352] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-2: Server lk version = 1 >[2012-06-01 04:29:35.930728] I [client-handshake.c:1636:select_server_supported_programs] 117-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:35.931406] I [client-handshake.c:1433:client_setvolume_cbk] 117-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:35.931438] I [client-handshake.c:1445:client_setvolume_cbk] 117-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:35.949923] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 117 >[2012-06-01 04:29:35.949980] I [client-handshake.c:453:client_set_lk_version_cbk] 117-master-client-3: Server lk version = 1 >[2012-06-01 04:29:37.947125] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:38.951052] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:38.951404] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:38.951435] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:38.951524] I [afr-common.c:3627:afr_notify] 118-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:38.951716] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-0: Server lk version = 1 >[2012-06-01 04:29:38.963022] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:38.963561] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:38.963653] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:38.963780] I [afr-common.c:3627:afr_notify] 118-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:29:38.964004] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-3: Server lk version = 1 >[2012-06-01 04:29:38.990305] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:38.990680] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:38.990710] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:38.991061] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-1: Server lk version = 1 >[2012-06-01 04:29:39.004725] I [client-handshake.c:1636:select_server_supported_programs] 118-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:39.005286] I [client-handshake.c:1433:client_setvolume_cbk] 118-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:39.005319] I [client-handshake.c:1445:client_setvolume_cbk] 118-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:39.014587] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 118 >[2012-06-01 04:29:39.014686] I [client-handshake.c:453:client_set_lk_version_cbk] 118-master-client-2: Server lk version = 1 >[2012-06-01 04:29:39.100069] I [io-cache.c:1549:check_cache_size_ok] 119-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:39.100206] I [io-cache.c:1549:check_cache_size_ok] 119-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:39.100311] I [quota.c:3085:quota_parse_limits] 119-master-quota: no "limit-set" option provided >[2012-06-01 04:29:39.108707] I [client.c:2142:notify] 119-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:39.114199] I [client.c:2142:notify] 119-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:39.119646] I [client.c:2142:notify] 119-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:39.124663] I [client.c:2142:notify] 119-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:39.132706] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:39.132891] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:39.133064] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:39.133200] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 119-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:39.834918] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 118-master-replicate-0: added root inode >[2012-06-01 04:29:39.836377] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 118-master-replicate-1: added root inode >[2012-06-01 04:29:39.907047] I [client.c:2151:notify] 114-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:39.907163] I [client.c:2151:notify] 114-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:39.907210] I [client.c:2151:notify] 114-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:39.907252] I [client.c:2151:notify] 114-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:29:39.907335] I [client.c:2090:client_rpc_notify] 114-master-client-0: disconnected >[2012-06-01 04:29:39.907381] I [client.c:2090:client_rpc_notify] 114-master-client-1: disconnected >[2012-06-01 04:29:39.907401] E [afr-common.c:3664:afr_notify] 114-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:29:39.907441] I [client.c:2090:client_rpc_notify] 114-master-client-2: disconnected >[2012-06-01 04:29:39.907507] I [client.c:2090:client_rpc_notify] 114-master-client-3: disconnected >[2012-06-01 04:29:39.907529] E [afr-common.c:3664:afr_notify] 114-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:29:41.386477] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:42.432810] I [io-cache.c:1549:check_cache_size_ok] 120-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:42.440015] I [io-cache.c:1549:check_cache_size_ok] 120-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:42.460986] I [client.c:2142:notify] 120-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:42.475183] I [client.c:2142:notify] 120-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:42.486314] I [client.c:2142:notify] 120-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:42.497810] I [client.c:2142:notify] 120-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:42.508436] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:42.510342] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:42.512064] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:42.512199] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 120-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:43.025386] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:43.025865] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:43.025915] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:43.026041] I [afr-common.c:3627:afr_notify] 119-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:43.026091] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-0: Server lk version = 1 >[2012-06-01 04:29:43.033171] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:43.033510] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:43.033542] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:43.034006] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-1: Server lk version = 1 >[2012-06-01 04:29:43.042369] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:43.042712] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:43.042745] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:43.042824] I [afr-common.c:3627:afr_notify] 119-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:43.042999] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-2: Server lk version = 1 >[2012-06-01 04:29:43.055119] I [client-handshake.c:1636:select_server_supported_programs] 119-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:43.055585] I [client-handshake.c:1433:client_setvolume_cbk] 119-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:43.055619] I [client-handshake.c:1445:client_setvolume_cbk] 119-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:43.108452] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 119 >[2012-06-01 04:29:43.108556] I [client-handshake.c:453:client_set_lk_version_cbk] 119-master-client-3: Server lk version = 1 >[2012-06-01 04:29:44.899631] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:46.057710] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:46.058038] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:46.058066] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:46.058161] I [afr-common.c:3627:afr_notify] 120-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:46.058221] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-0: Server lk version = 1 >[2012-06-01 04:29:46.065861] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:46.066753] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:46.066798] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:46.067164] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-1: Server lk version = 1 >[2012-06-01 04:29:46.090084] I [io-cache.c:1549:check_cache_size_ok] 121-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:46.090225] I [io-cache.c:1549:check_cache_size_ok] 121-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:46.090317] I [quota.c:3085:quota_parse_limits] 121-master-quota: no "limit-set" option provided >[2012-06-01 04:29:46.105877] I [client.c:2142:notify] 121-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:46.120491] I [client.c:2142:notify] 121-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:46.136090] I [client.c:2142:notify] 121-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:46.157560] I [client.c:2142:notify] 121-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:46.178186] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:46.178585] I [client-handshake.c:1636:select_server_supported_programs] 120-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:46.179054] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:46.179089] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:46.179336] I [afr-common.c:3627:afr_notify] 120-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:46.179491] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-2: Server lk version = 1 >[2012-06-01 04:29:46.179556] I [client-handshake.c:1433:client_setvolume_cbk] 120-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:46.179587] I [client-handshake.c:1445:client_setvolume_cbk] 120-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:46.208130] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 120 >[2012-06-01 04:29:46.208221] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:46.208328] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:46.208409] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:46.208502] I [client-handshake.c:453:client_set_lk_version_cbk] 120-master-client-3: Server lk version = 1 >[2012-06-01 04:29:46.208585] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 121-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:50.108397] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:50.108771] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:50.108804] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:50.108892] I [afr-common.c:3627:afr_notify] 121-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:50.109032] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-0: Server lk version = 1 >[2012-06-01 04:29:50.135885] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:50.136349] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:50.136380] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:50.136986] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-1: Server lk version = 1 >[2012-06-01 04:29:50.147688] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:50.148071] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:50.148104] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:50.148177] I [afr-common.c:3627:afr_notify] 121-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:50.148787] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-2: Server lk version = 1 >[2012-06-01 04:29:50.210320] I [client-handshake.c:1636:select_server_supported_programs] 121-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:50.211478] I [client-handshake.c:1433:client_setvolume_cbk] 121-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:50.211510] I [client-handshake.c:1445:client_setvolume_cbk] 121-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:50.253177] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 121 >[2012-06-01 04:29:50.253276] I [client-handshake.c:453:client_set_lk_version_cbk] 121-master-client-3: Server lk version = 1 >[2012-06-01 04:29:53.263495] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:54.344286] I [io-cache.c:1549:check_cache_size_ok] 122-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:54.344360] I [io-cache.c:1549:check_cache_size_ok] 122-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:54.380468] I [client.c:2142:notify] 122-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:54.390741] I [client.c:2142:notify] 122-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:54.393604] I [client.c:2142:notify] 122-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:54.401937] I [client.c:2142:notify] 122-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:54.417263] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:54.417331] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:54.417410] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:54.417466] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 122-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:56.768421] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:29:57.937034] I [io-cache.c:1549:check_cache_size_ok] 123-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:29:57.937120] I [io-cache.c:1549:check_cache_size_ok] 123-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:29:57.937186] I [quota.c:3085:quota_parse_limits] 123-master-quota: no "limit-set" option provided >[2012-06-01 04:29:57.952369] I [client.c:2142:notify] 123-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:57.968731] I [client.c:2142:notify] 123-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:57.978730] I [client.c:2142:notify] 123-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:29:57.985146] I [client.c:2142:notify] 123-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:29:57.997898] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:29:57.998002] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:29:57.998090] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:29:57.998177] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 123-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:29:58.216379] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:58.216716] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:29:58.216739] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:58.216800] I [afr-common.c:3627:afr_notify] 122-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:29:58.217976] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-0: Server lk version = 1 >[2012-06-01 04:29:58.229045] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:58.229313] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:29:58.229336] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:58.231326] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:58.231557] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:29:58.231580] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:58.231662] I [afr-common.c:3627:afr_notify] 122-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:29:58.231766] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-2: Server lk version = 1 >[2012-06-01 04:29:58.232332] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-1: Server lk version = 1 >[2012-06-01 04:29:58.242674] I [client-handshake.c:1636:select_server_supported_programs] 122-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:29:58.243009] I [client-handshake.c:1433:client_setvolume_cbk] 122-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:29:58.243036] I [client-handshake.c:1445:client_setvolume_cbk] 122-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:29:58.258824] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 122 >[2012-06-01 04:29:58.258887] I [client-handshake.c:453:client_set_lk_version_cbk] 122-master-client-3: Server lk version = 1 >[2012-06-01 04:30:00.604384] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 122-master-replicate-0: added root inode >[2012-06-01 04:30:00.626505] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 122-master-replicate-1: added root inode >[2012-06-01 04:30:00.627675] I [client.c:2151:notify] 118-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 04:30:00.627726] I [client.c:2151:notify] 118-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 04:30:00.627766] I [client.c:2151:notify] 118-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 04:30:00.627796] I [client.c:2151:notify] 118-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 04:30:00.631081] I [client.c:2090:client_rpc_notify] 118-master-client-0: disconnected >[2012-06-01 04:30:00.631120] I [client.c:2090:client_rpc_notify] 118-master-client-1: disconnected >[2012-06-01 04:30:00.631133] E [afr-common.c:3664:afr_notify] 118-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:30:00.631155] I [client.c:2090:client_rpc_notify] 118-master-client-2: disconnected >[2012-06-01 04:30:00.631183] I [client.c:2090:client_rpc_notify] 118-master-client-3: disconnected >[2012-06-01 04:30:00.631194] E [afr-common.c:3664:afr_notify] 118-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 04:30:01.245876] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:01.246090] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:30:01.246109] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:01.246157] I [afr-common.c:3627:afr_notify] 123-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:30:01.246238] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-0: Server lk version = 1 >[2012-06-01 04:30:01.253744] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:01.253977] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:30:01.253997] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:01.254216] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-1: Server lk version = 1 >[2012-06-01 04:30:01.274372] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:01.274810] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:30:01.274836] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:01.274908] I [afr-common.c:3627:afr_notify] 123-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:30:01.275001] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-2: Server lk version = 1 >[2012-06-01 04:30:01.292925] I [client-handshake.c:1636:select_server_supported_programs] 123-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:01.293153] I [client-handshake.c:1433:client_setvolume_cbk] 123-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:30:01.293175] I [client-handshake.c:1445:client_setvolume_cbk] 123-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:01.299815] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 123 >[2012-06-01 04:30:01.299933] I [client-handshake.c:453:client_set_lk_version_cbk] 123-master-client-3: Server lk version = 1 >[2012-06-01 04:30:05.646089] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:30:06.700797] I [io-cache.c:1549:check_cache_size_ok] 124-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:30:06.700912] I [io-cache.c:1549:check_cache_size_ok] 124-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:30:06.716126] I [client.c:2142:notify] 124-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:06.722277] I [client.c:2142:notify] 124-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:06.725529] I [client.c:2142:notify] 124-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:06.730414] I [client.c:2142:notify] 124-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:30:06.734999] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:30:06.735103] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:30:06.735206] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:30:06.735301] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 124-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:30:10.304654] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:10.304914] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:30:10.304936] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:10.304989] I [afr-common.c:3627:afr_notify] 124-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:30:10.305109] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-0: Server lk version = 1 >[2012-06-01 04:30:10.309793] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:10.310012] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:30:10.310033] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:10.310221] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-1: Server lk version = 1 >[2012-06-01 04:30:10.317049] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:10.317264] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:30:10.317285] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:10.317335] I [afr-common.c:3627:afr_notify] 124-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:30:10.317425] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-2: Server lk version = 1 >[2012-06-01 04:30:10.323722] I [client-handshake.c:1636:select_server_supported_programs] 124-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:10.324101] I [client-handshake.c:1433:client_setvolume_cbk] 124-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:30:10.324121] I [client-handshake.c:1445:client_setvolume_cbk] 124-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:10.336215] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 124 >[2012-06-01 04:30:10.336290] I [client-handshake.c:453:client_set_lk_version_cbk] 124-master-client-3: Server lk version = 1 >[2012-06-01 04:30:19.352031] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:30:25.431146] I [io-cache.c:1549:check_cache_size_ok] 125-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:30:25.431273] I [io-cache.c:1549:check_cache_size_ok] 125-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:30:25.431393] I [quota.c:3085:quota_parse_limits] 125-master-quota: no "limit-set" option provided >[2012-06-01 04:30:25.438730] I [client.c:2142:notify] 125-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:25.445169] I [client.c:2142:notify] 125-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:25.451797] I [client.c:2142:notify] 125-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:25.457021] I [client.c:2142:notify] 125-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:30:25.462742] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:30:25.463013] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:30:25.463088] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:30:25.463173] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 125-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:30:27.599222] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:30:28.633723] I [io-cache.c:1549:check_cache_size_ok] 126-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:30:28.633840] I [io-cache.c:1549:check_cache_size_ok] 126-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:30:28.658307] I [client.c:2142:notify] 126-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:28.672352] I [client.c:2142:notify] 126-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:28.680491] I [client.c:2142:notify] 126-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:28.700291] I [client.c:2142:notify] 126-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:30:28.707247] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:30:28.707380] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:30:28.707504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:30:28.707648] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 126-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:30:29.346100] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:29.346623] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:30:29.346655] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:29.346766] I [afr-common.c:3627:afr_notify] 125-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-06-01 04:30:29.346942] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-1: Server lk version = 1 >[2012-06-01 04:30:29.351972] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:29.352279] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:30:29.352309] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:29.352718] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-0: Server lk version = 1 >[2012-06-01 04:30:29.358768] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:29.359238] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:30:29.359274] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:29.359377] I [afr-common.c:3627:afr_notify] 125-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-06-01 04:30:29.359423] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-3: Server lk version = 1 >[2012-06-01 04:30:29.364515] I [client-handshake.c:1636:select_server_supported_programs] 125-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:29.365048] I [client-handshake.c:1433:client_setvolume_cbk] 125-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:30:29.365082] I [client-handshake.c:1445:client_setvolume_cbk] 125-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:29.372940] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 125 >[2012-06-01 04:30:29.373084] I [client-handshake.c:453:client_set_lk_version_cbk] 125-master-client-2: Server lk version = 1 >[2012-06-01 04:30:32.394487] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:32.394753] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:30:32.394773] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:32.394819] I [afr-common.c:3627:afr_notify] 126-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:30:32.394912] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-0: Server lk version = 1 >[2012-06-01 04:30:32.399612] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:32.399913] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:30:32.399934] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:32.400275] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-1: Server lk version = 1 >[2012-06-01 04:30:32.443187] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:32.446198] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:30:32.446232] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:32.446284] I [afr-common.c:3627:afr_notify] 126-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:30:32.446376] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-2: Server lk version = 1 >[2012-06-01 04:30:32.476172] I [client-handshake.c:1636:select_server_supported_programs] 126-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:32.476483] I [client-handshake.c:1433:client_setvolume_cbk] 126-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:30:32.476514] I [client-handshake.c:1445:client_setvolume_cbk] 126-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:32.493553] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 126 >[2012-06-01 04:30:32.493654] I [client-handshake.c:453:client_set_lk_version_cbk] 126-master-client-3: Server lk version = 1 >[2012-06-01 04:30:41.318300] I [glusterfsd-mgmt.c:64:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2012-06-01 04:30:47.507290] I [io-cache.c:1549:check_cache_size_ok] 127-master-quick-read: Max cache size is 2104233984 >[2012-06-01 04:30:47.507405] I [io-cache.c:1549:check_cache_size_ok] 127-master-io-cache: Max cache size is 2104233984 >[2012-06-01 04:30:47.507504] I [quota.c:3085:quota_parse_limits] 127-master-quota: no "limit-set" option provided >[2012-06-01 04:30:47.521566] I [client.c:2142:notify] 127-master-client-0: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:47.526308] I [client.c:2142:notify] 127-master-client-1: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:47.530910] I [client.c:2142:notify] 127-master-client-2: parent translators are ready, attempting connect on transport >[2012-06-01 04:30:47.536805] I [client.c:2142:notify] 127-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.151 > 4: option remote-subvolume /exportdir/d1 > 5: option transport-type tcp > 6: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 7: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.151 > 13: option remote-subvolume /exportdir/d2 > 14: option transport-type tcp > 15: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 16: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.151 > 22: option remote-subvolume /exportdir/d3 > 23: option transport-type tcp > 24: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 25: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.151 > 31: option remote-subvolume /exportdir/d4 > 32: option transport-type tcp > 33: option username ab033ab9-6b51-4547-a7b5-41c06850a6e7 > 34: option password cb4e4265-d329-4ee5-85d3-ae6c76565f06 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-quota > 53: type features/quota > 54: option timeout 0 > 55: subvolumes master-dht > 56: end-volume > 57: > 58: volume master-write-behind > 59: type performance/write-behind > 60: subvolumes master-quota > 61: end-volume > 62: > 63: volume master-read-ahead > 64: type performance/read-ahead > 65: subvolumes master-write-behind > 66: end-volume > 67: > 68: volume master-io-cache > 69: type performance/io-cache > 70: subvolumes master-read-ahead > 71: end-volume > 72: > 73: volume master-quick-read > 74: type performance/quick-read > 75: subvolumes master-io-cache > 76: end-volume > 77: > 78: volume master-md-cache > 79: type performance/md-cache > 80: subvolumes master-quick-read > 81: end-volume > 82: > 83: volume master > 84: type debug/io-stats > 85: option latency-measurement off > 86: option count-fop-hits off > 87: subvolumes master-md-cache > 88: end-volume > >+------------------------------------------------------------------------------+ >[2012-06-01 04:30:47.546250] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-0: changing port to 24009 (from 0) >[2012-06-01 04:30:47.546383] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-1: changing port to 24010 (from 0) >[2012-06-01 04:30:47.546504] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-2: changing port to 24011 (from 0) >[2012-06-01 04:30:47.546661] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 127-master-client-3: changing port to 24012 (from 0) >[2012-06-01 04:30:51.497964] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-0: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:51.498317] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-0: Connected to 172.17.251.151:24009, attached to remote volume '/exportdir/d1'. >[2012-06-01 04:30:51.498346] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:51.498459] I [afr-common.c:3627:afr_notify] 127-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-06-01 04:30:51.498506] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-0: Server lk version = 1 >[2012-06-01 04:30:51.504268] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-1: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:51.504603] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-1: Connected to 172.17.251.151:24010, attached to remote volume '/exportdir/d2'. >[2012-06-01 04:30:51.504631] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:51.504817] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-1: Server lk version = 1 >[2012-06-01 04:30:51.509815] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-2: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:51.510520] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-2: Connected to 172.17.251.151:24011, attached to remote volume '/exportdir/d3'. >[2012-06-01 04:30:51.510550] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:51.510645] I [afr-common.c:3627:afr_notify] 127-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-06-01 04:30:51.510801] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-2: Server lk version = 1 >[2012-06-01 04:30:51.517811] I [client-handshake.c:1636:select_server_supported_programs] 127-master-client-3: Using Program GlusterFS 3.3.0qa45, Num (1298437), Version (330) >[2012-06-01 04:30:51.518437] I [client-handshake.c:1433:client_setvolume_cbk] 127-master-client-3: Connected to 172.17.251.151:24012, attached to remote volume '/exportdir/d4'. >[2012-06-01 04:30:51.518469] I [client-handshake.c:1445:client_setvolume_cbk] 127-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-06-01 04:30:51.524304] I [fuse-bridge.c:4193:fuse_graph_setup] 0-fuse: switched to graph 127 >[2012-06-01 04:30:51.524954] I [client-handshake.c:453:client_set_lk_version_cbk] 127-master-client-3: Server lk version = 1 >[2012-06-01 05:00:29.938055] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 127-master-replicate-0: added root inode >[2012-06-01 05:00:29.941310] I [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 127-master-replicate-1: added root inode >[2012-06-01 05:00:29.944795] I [client.c:2151:notify] 122-master-client-0: current graph is no longer active, destroying rpc_client >[2012-06-01 05:00:29.944908] I [client.c:2151:notify] 122-master-client-1: current graph is no longer active, destroying rpc_client >[2012-06-01 05:00:29.945175] I [client.c:2090:client_rpc_notify] 122-master-client-0: disconnected >[2012-06-01 05:00:29.945190] I [client.c:2151:notify] 122-master-client-2: current graph is no longer active, destroying rpc_client >[2012-06-01 05:00:29.945234] I [client.c:2090:client_rpc_notify] 122-master-client-1: disconnected >[2012-06-01 05:00:29.945257] E [afr-common.c:3664:afr_notify] 122-master-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2012-06-01 05:00:29.945304] I [client.c:2151:notify] 122-master-client-3: current graph is no longer active, destroying rpc_client >[2012-06-01 05:00:29.945523] I [client.c:2090:client_rpc_notify] 122-master-client-2: disconnected >[2012-06-01 05:00:29.945892] I [client.c:2090:client_rpc_notify] 122-master-client-3: disconnected >[2012-06-01 05:00:29.945920] E [afr-common.c:3664:afr_notify] 122-master-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 827405
: 588419