Attachment 577300 Details for Bug 812314 – maser-rebalance log file .

maser-rebalance log file .

master-rebalance.log (text/x-log), 432.62 KB, created by Vijaykumar Koppad on 2012-04-13 11:05:12 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Vijaykumar Koppad

Created: 2012-04-13 11:05:12 UTC

Size: 432.62 KB

patch

obsolete

>[2012-04-11 21:39:05.569837] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-11 21:39:10.571881] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-11 21:39:10.573190] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-11 21:39:10.577997] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-11 21:39:10.578029] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1'
>[2012-04-11 21:39:10.578050] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:39:10.578068] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:39:10.578086] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:39:10.578293] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-11 21:39:10.578405] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-11 21:39:10.589200] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-11 21:39:10.593997] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-11 21:39:10.598650] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-11 21:39:10.603208] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-11 21:39:10.607810] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-11 21:39:10.612435] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 73: end-volume
> 74: 
> 75: volume master-write-behind
> 76:     type performance/write-behind
> 77:     subvolumes master-dht
> 78: end-volume
> 79: 
> 80: volume master-read-ahead
> 81:     type performance/read-ahead
> 82:     subvolumes master-write-behind
> 83: end-volume
> 84: 
> 85: volume master-io-cache
> 86:     type performance/io-cache
> 87:     subvolumes master-read-ahead
> 88: end-volume
> 89: 
> 90: volume master-quick-read
> 91:     type performance/quick-read
> 92:     subvolumes master-io-cache
> 93: end-volume
> 94: 
> 95: volume master-md-cache
> 96:     type performance/md-cache
> 97:     subvolumes master-quick-read
> 98: end-volume
> 99: 
>100: volume master
>101:     type debug/io-stats
>102:     option latency-measurement off
>103:     option count-fop-hits off
>104:     subvolumes master-md-cache
>105: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-11 21:39:10.619113] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-11 21:39:10.619275] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-11 21:39:10.619422] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-11 21:39:10.619583] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-11 21:39:10.619726] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24013 (from 0)
>[2012-04-11 21:39:10.619904] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24014 (from 0)
>[2012-04-11 21:39:10.620059] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-11 21:39:10.620116] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-11 21:39:10.620158] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-11 21:39:10.620197] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-11 21:39:10.620236] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-11 21:39:10.620274] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-11 21:39:14.590132] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-11 21:39:14.590347] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.590790] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-11 21:39:14.590825] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.591050] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-04-11 21:39:14.591244] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-11 21:39:14.595991] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-11 21:39:14.596175] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.596605] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-11 21:39:14.596638] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.597638] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-11 21:39:14.602487] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-11 21:39:14.602655] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.603047] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-11 21:39:14.603083] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.603165] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-11 21:39:14.603368] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-11 21:39:14.608123] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-11 21:39:14.608303] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.608658] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-11 21:39:14.608691] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.609484] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-11 21:39:14.614279] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-11 21:39:14.614445] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.614840] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24013, attached to remote volume '/exportdir/master/d5'.
>[2012-04-11 21:39:14.614875] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.614956] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-11 21:39:14.615140] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-11 21:39:14.619909] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-11 21:39:14.620073] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:39:14.620453] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24014, attached to remote volume '/exportdir/master/d6'.
>[2012-04-11 21:39:14.620486] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:39:14.621319] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-11 21:39:14.641034] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-11 21:39:14.644048] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-11 21:39:14.644203] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-11 21:39:14.644460] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-11 21:39:14.652888] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-11 21:39:14.671114] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-11 21:39:14.680396] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-11 21:39:14.693968] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:15.938269] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 1
>[2012-04-11 21:39:17.994659] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:17.994699] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:17.994769] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:17.994798] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:17.995336] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:18.000429] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:18.000466] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:18.009214] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:19.553575] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 100000000, lookups: 2
>[2012-04-11 21:39:21.749250] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 100000000, lookups: 2
>[2012-04-11 21:39:22.366777] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:22.366814] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:22.366854] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:22.366876] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:22.370247] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:22.378208] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:22.378244] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:22.389262] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:23.710172] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 200000000, lookups: 3
>[2012-04-11 21:39:25.986970] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 200000000, lookups: 3
>[2012-04-11 21:39:26.157536] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:26.157575] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:26.157612] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:26.157634] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:26.162106] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:26.173790] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:26.173832] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:26.261196] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-11 21:39:26.290715] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-11 21:39:26.324693] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:27.979940] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 14
>[2012-04-11 21:39:29.654944] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 14
>[2012-04-11 21:39:29.843913] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:29.843950] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:29.843988] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:29.844011] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:29.844453] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:29.849726] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:29.849793] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:29.854900] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:31.577233] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 400000000, lookups: 15
>[2012-04-11 21:39:33.286935] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 400000000, lookups: 15
>[2012-04-11 21:39:34.254543] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:34.254580] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:34.254618] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:34.254641] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:34.255108] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:34.263628] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:34.263667] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:34.270923] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:37.014401] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 5, size: 500000000, lookups: 16
>[2012-04-11 21:39:38.002869] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:38.002908] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:38.002947] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:38.002970] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:38.003398] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:38.008594] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:38.008632] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:38.037804] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-11 21:39:38.048777] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-11 21:39:38.066058] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:38.863035] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 27
>[2012-04-11 21:39:41.341718] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 27
>[2012-04-11 21:39:41.773887] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:41.773927] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:41.773969] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:41.773992] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:41.774446] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:41.779366] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:41.779403] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:41.784379] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:43.204921] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 28
>[2012-04-11 21:39:44.622875] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 28
>[2012-04-11 21:39:45.041710] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:45.041757] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:45.041785] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:45.041799] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:45.042064] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:45.045656] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:45.045681] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:45.049330] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:48.021014] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:48.021041] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:39:48.021064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:48.021077] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:39:48.021347] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:39:48.024378] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:48.024400] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:39:48.042324] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-11 21:39:48.048213] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-11 21:39:48.062101] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:50.811403] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:50.811430] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:39:50.811454] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:50.811467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:39:50.811731] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:39:50.814720] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:50.814770] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:50.818091] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:53.456307] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:53.456333] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:39:53.456355] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:53.456377] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:39:53.456626] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:39:53.459492] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:53.459513] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:53.462777] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:56.194607] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:56.194633] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:39:56.194656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:56.194669] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:39:56.194942] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:39:56.197853] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:56.197875] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:56.217411] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-11 21:39:56.227532] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-11 21:39:56.245717] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:58.914576] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:58.914603] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:39:58.914626] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:39:58.914640] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:39:58.914923] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:39:58.918054] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:39:58.918076] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:39:58.921340] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:01.899675] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:01.899702] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:01.899725] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:01.899758] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:01.900018] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:01.902980] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:01.903002] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:01.906349] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:04.816933] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:04.816960] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:04.816983] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:04.816996] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:04.817250] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:04.820230] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:04.820251] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:04.835495] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-11 21:40:04.840857] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-11 21:40:04.854492] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:07.711953] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:07.711981] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:07.712005] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:07.712018] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:07.712295] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:07.715255] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:07.715277] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:07.719190] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:10.868821] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:10.868859] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:10.868898] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:10.868920] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:10.869344] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:10.874287] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:10.874323] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:10.879292] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:14.458031] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:14.458070] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:14.458108] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:14.458130] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:14.458580] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:14.463548] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:14.463585] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:14.485259] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-11 21:40:14.494716] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-11 21:40:14.509638] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:17.800729] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:17.800799] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:40:17.800841] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:17.800873] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:40:17.801302] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:40:17.806319] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:17.806356] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:17.811426] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:21.118549] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:21.118588] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:40:21.118626] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:21.118648] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:40:21.119125] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:40:21.128908] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:21.128951] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:21.139075] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:24.349520] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:24.349572] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 21:40:24.349627] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:24.349653] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 21:40:24.350216] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 21:40:24.358303] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:24.358338] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 21:40:24.389809] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-11 21:40:24.399070] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-11 21:40:24.422680] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:27.808676] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:27.808715] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:27.808798] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:27.808826] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:27.809257] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:27.814241] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:27.814277] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:27.820215] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:31.002968] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:31.003005] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:31.003045] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:31.003068] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:31.003502] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:31.008589] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:31.008627] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:31.014210] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:34.320123] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:34.320160] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 21:40:34.320197] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:40:34.320218] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 21:40:34.320629] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 21:40:34.328337] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:40:34.328371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 21:40:34.359478] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 104
>[2012-04-11 21:40:34.360009] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-11 21:45:07.471622] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-11 21:45:12.473586] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-11 21:45:12.473661] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-11 21:45:12.560459] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-11 21:45:12.560485] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5'
>[2012-04-11 21:45:12.560497] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:45:12.560507] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:45:12.560517] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-11 21:45:12.560643] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-11 21:45:12.560711] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-11 21:45:12.560825] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2
>[2012-04-11 21:45:13.004366] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-11 21:45:13.007066] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-11 21:45:13.009527] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-11 21:45:13.011948] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-11 21:45:13.014344] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-11 21:45:13.016827] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     option decommissioned-bricks master-replicate-2
> 73:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 74: end-volume
> 75: 
> 76: volume master-write-behind
> 77:     type performance/write-behind
> 78:     subvolumes master-dht
> 79: end-volume
> 80: 
> 81: volume master-read-ahead
> 82:     type performance/read-ahead
> 83:     subvolumes master-write-behind
> 84: end-volume
> 85: 
> 86: volume master-io-cache
> 87:     type performance/io-cache
> 88:     subvolumes master-read-ahead
> 89: end-volume
> 90: 
> 91: volume master-quick-read
> 92:     type performance/quick-read
> 93:     subvolumes master-io-cache
> 94: end-volume
> 95: 
> 96: volume master-md-cache
> 97:     type performance/md-cache
> 98:     subvolumes master-quick-read
> 99: end-volume
>100: 
>101: volume master
>102:     type debug/io-stats
>103:     option latency-measurement off
>104:     option count-fop-hits off
>105:     subvolumes master-md-cache
>106: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-11 21:45:13.020553] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-11 21:45:13.020663] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-11 21:45:13.020771] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-11 21:45:13.020860] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-11 21:45:13.020946] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24013 (from 0)
>[2012-04-11 21:45:13.021031] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24014 (from 0)
>[2012-04-11 21:45:13.021124] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-11 21:45:13.021159] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-11 21:45:13.021183] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-11 21:45:13.021205] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-11 21:45:13.021227] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-11 21:45:13.021248] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-11 21:45:16.487882] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-11 21:45:16.488084] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.493173] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-11 21:45:16.493335] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.498904] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-11 21:45:16.499068] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.502339] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-11 21:45:16.502375] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.611686] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-11 21:45:16.611849] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-11 21:45:16.611881] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.612185] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-04-11 21:45:16.612285] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-11 21:45:16.612316] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.613321] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-11 21:45:16.613377] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-11 21:45:16.613429] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-11 21:45:16.613625] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-11 21:45:16.613728] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.614188] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-11 21:45:16.614221] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.615154] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-11 21:45:16.617941] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-11 21:45:16.618004] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.618367] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24013, attached to remote volume '/exportdir/master/d5'.
>[2012-04-11 21:45:16.618399] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.618656] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-11 21:45:16.618716] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-11 21:45:16.621189] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-11 21:45:16.621367] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 21:45:16.621938] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24014, attached to remote volume '/exportdir/master/d6'.
>[2012-04-11 21:45:16.621984] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 21:45:16.623267] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-11 21:45:16.706355] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-11 21:45:16.709613] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-11 21:45:16.709794] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-11 21:45:16.710044] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-11 21:45:16.718646] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-11 21:45:16.737361] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-11 21:45:16.746951] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-11 21:45:16.794287] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:20.909698] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:20.909767] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:20.909816] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:20.909839] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:20.910267] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:20.917556] W [dht-common.c:945:dht_lookup_everywhere_cbk] 0-master-dht: multiple subvolumes (master-replicate-0 and master-replicate-1) have file /large1/file0 (preferably rename the file in the backend, and do a fresh lookup)
>[2012-04-11 21:45:20.918689] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:20.918726] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:20.927645] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large1/file1 on master-replicate-1
>[2012-04-11 21:45:20.940863] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:24.899449] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:24.899475] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:24.899498] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:24.899511] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:24.899781] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:24.902833] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:24.902855] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:24.905000] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large1/file2 on master-replicate-1
>[2012-04-11 21:45:24.913856] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:28.549245] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:28.549275] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:28.549298] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:28.549322] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:28.549582] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:28.552528] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:28.552549] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:28.561353] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-11 21:45:28.566565] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-11 21:45:28.584566] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:28.820456] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 18
>[2012-04-11 21:45:31.343510] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:31.343537] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:31.343560] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:31.343573] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:31.343846] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:31.346687] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:31.346708] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:31.349692] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:34.124531] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:34.124558] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:34.124580] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:34.124593] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:34.126024] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:34.133147] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:34.133179] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:34.136384] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:37.089609] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:37.089636] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:37.089659] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:37.089672] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:37.089942] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:37.092907] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:37.092930] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:37.099718] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-11 21:45:37.104967] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-11 21:45:37.120711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:40.752657] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:40.752685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:40.752709] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:40.752722] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:40.753240] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:40.759474] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:40.759496] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:40.769529] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:43.418196] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:43.418224] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:43.418252] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:43.418266] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:43.418539] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:43.421523] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:43.421545] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:43.427171] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:46.229057] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:46.229084] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:46.229106] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:46.229119] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:46.229381] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:46.232324] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:46.232346] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:45:46.239145] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-11 21:45:46.244511] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-11 21:45:46.260170] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:48.967314] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:48.967341] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:48.967364] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:48.967397] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:48.967682] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:48.970728] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:48.970767] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:48.982688] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:51.980446] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:51.980472] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:51.980495] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:51.980507] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:51.980791] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:51.983723] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:51.983762] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:51.989372] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:54.589678] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:54.589704] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:54.589727] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:54.589761] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:54.590018] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:54.598571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:54.598594] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:54.610921] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-11 21:45:54.620383] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-11 21:45:54.655676] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:57.384754] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:57.384781] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:45:57.384805] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:45:57.384818] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:45:57.385071] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:45:57.388018] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:45:57.388040] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:45:57.393582] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:00.091807] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:00.091834] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:00.091855] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:00.091874] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:00.092130] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:00.095313] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:00.095334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:00.101119] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:02.910504] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:02.910532] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:02.910555] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:02.910568] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:02.910855] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:02.913836] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:02.913857] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:02.920650] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-11 21:46:02.925950] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-11 21:46:02.941791] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:05.660484] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:05.660510] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:05.660534] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:05.660547] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:05.660831] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:05.663795] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:05.663816] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:05.669644] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:08.439593] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:08.439619] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:08.439654] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:08.439667] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:08.439938] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:08.442888] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:08.442914] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:08.448679] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:11.144548] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:11.144574] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:11.144607] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:11.144621] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:11.144888] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:11.147807] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:11.147833] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:11.154436] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-11 21:46:11.159682] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-11 21:46:11.175727] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:14.133404] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:14.133431] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:14.133453] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:14.133466] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:14.133724] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:14.136575] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:14.136606] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:14.142057] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:14.662099] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 69
>[2012-04-11 21:46:16.599375] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 69
>[2012-04-11 21:46:16.998442] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:16.998467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:16.998490] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:16.998503] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:16.998771] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:17.001638] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:17.001659] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:17.007097] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:18.124543] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 70
>[2012-04-11 21:46:19.450415] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 70
>[2012-04-11 21:46:19.758251] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:19.758278] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:19.758301] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:19.758314] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:19.758573] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:19.761511] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:19.761532] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-11 21:46:19.768246] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-11 21:46:19.773361] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-11 21:46:19.788479] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:22.438477] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:22.438503] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:22.438525] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:22.438538] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:22.438804] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:22.441672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:22.441693] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:22.447401] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:25.183283] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:25.183309] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:25.183332] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:25.183345] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:25.183601] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:25.186499] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:25.186520] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:25.192017] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:27.943389] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:27.943416] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-11 21:46:27.943438] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 21:46:27.943451] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-11 21:46:27.943714] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-11 21:46:27.946638] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 21:46:27.946660] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-11 21:46:27.953202] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 80
>[2012-04-11 21:46:27.953518] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-11 22:04:07.447208] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-11 22:04:12.448572] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-11 22:04:12.448631] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-11 22:04:12.450774] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-11 22:04:12.450804] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1'
>[2012-04-11 22:04:12.450817] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-11 22:04:12.450828] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-11 22:04:12.450838] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-11 22:04:12.450983] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-11 22:04:12.451052] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-11 22:04:12.460273] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-11 22:04:12.462966] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-11 22:04:12.465318] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-11 22:04:12.467676] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-11 22:04:12.470120] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-11 22:04:12.472481] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 73: end-volume
> 74: 
> 75: volume master-write-behind
> 76:     type performance/write-behind
> 77:     subvolumes master-dht
> 78: end-volume
> 79: 
> 80: volume master-read-ahead
> 81:     type performance/read-ahead
> 82:     subvolumes master-write-behind
> 83: end-volume
> 84: 
> 85: volume master-io-cache
> 86:     type performance/io-cache
> 87:     subvolumes master-read-ahead
> 88: end-volume
> 89: 
> 90: volume master-quick-read
> 91:     type performance/quick-read
> 92:     subvolumes master-io-cache
> 93: end-volume
> 94: 
> 95: volume master-md-cache
> 96:     type performance/md-cache
> 97:     subvolumes master-quick-read
> 98: end-volume
> 99: 
>100: volume master
>101:     type debug/io-stats
>102:     option latency-measurement off
>103:     option count-fop-hits off
>104:     subvolumes master-md-cache
>105: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-11 22:04:12.476177] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-11 22:04:12.476280] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-11 22:04:12.476377] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-11 22:04:12.476464] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-11 22:04:12.476549] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24015 (from 0)
>[2012-04-11 22:04:12.476635] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24016 (from 0)
>[2012-04-11 22:04:12.476728] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-11 22:04:12.476784] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-11 22:04:12.476809] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-11 22:04:12.476831] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-11 22:04:12.476854] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-11 22:04:12.476875] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-11 22:04:16.466525] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-11 22:04:16.466725] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.467157] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-11 22:04:16.467188] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.467512] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-04-11 22:04:16.467696] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-11 22:04:16.472312] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-11 22:04:16.472466] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.472862] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-11 22:04:16.472895] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.473777] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-11 22:04:16.478607] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-11 22:04:16.478797] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.479150] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-11 22:04:16.479183] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.479263] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-11 22:04:16.479439] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-11 22:04:16.484162] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-11 22:04:16.484325] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.484701] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-11 22:04:16.484766] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.485536] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-11 22:04:16.490390] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-11 22:04:16.490584] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.490996] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24015, attached to remote volume '/exportdir/master/d5'.
>[2012-04-11 22:04:16.491030] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.491113] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-11 22:04:16.491297] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-11 22:04:16.496030] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-11 22:04:16.496194] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-11 22:04:16.496576] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24016, attached to remote volume '/exportdir/master/d6'.
>[2012-04-11 22:04:16.496609] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-11 22:04:16.497440] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-11 22:04:16.514402] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-11 22:04:16.517774] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-11 22:04:16.517923] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-11 22:04:16.518165] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-11 22:04:16.526840] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-11 22:04:16.545912] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-11 22:04:16.560565] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-11 22:04:16.573629] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:20.292580] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:20.292606] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:20.292629] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:20.292642] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:20.292920] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:20.295857] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:20.295878] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:20.297905] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:24.797634] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:24.797672] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:24.797711] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:24.797763] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:24.798197] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:24.807888] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:24.807923] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:24.831998] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:28.379440] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:28.379467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:28.379490] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:28.379507] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:28.379786] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:28.382967] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:28.382990] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:28.385943] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:32.177437] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:32.177476] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:32.177514] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:32.177537] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:32.178026] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:32.183026] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:32.183063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:32.188009] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:36.083605] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:36.083632] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:36.083655] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:36.083669] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:36.083956] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:36.086913] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:36.086935] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:36.090330] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:39.037660] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:39.037688] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:39.037712] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:39.037726] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:39.038006] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:39.041008] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:39.041030] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:04:39.046153] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file3: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:41.732430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:41.732458] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:04:41.732481] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:41.732493] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:04:41.732759] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file3: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:04:41.735672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:41.735697] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file3 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:41.738952] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file4: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:44.409656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:44.409684] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:04:44.409708] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:44.409722] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:04:44.409997] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file4: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:04:44.412982] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:44.413005] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file4 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:44.416469] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file7: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:47.022877] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:47.022904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:04:47.022926] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:47.022940] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:04:47.023189] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file7: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:04:47.026112] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:47.026133] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file7 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:47.029387] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file9: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:49.742147] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:49.742174] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:04:49.742196] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:49.742209] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:04:49.742454] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file9: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:04:49.745328] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:49.745349] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file9 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:04:49.757494] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-11 22:04:49.762629] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-11 22:04:49.773372] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:52.438035] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:52.438061] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:52.438083] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:52.438096] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:52.438352] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:52.441936] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:52.441957] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:52.445308] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:55.182358] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:55.182383] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:55.182415] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:55.182429] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:55.182675] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:55.185611] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:55.185632] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:55.188917] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:57.887546] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:57.887588] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:04:57.887616] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:04:57.887634] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:04:57.887921] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:04:57.891530] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:04:57.891552] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:04:57.905707] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-11 22:04:57.910962] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-11 22:04:57.916853] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large3/file5 on master-replicate-1
>[2012-04-11 22:04:57.922803] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:00.644687] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:00.644727] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:00.644802] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:00.644827] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:00.645252] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:00.650462] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:00.650512] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:00.655997] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:03.398381] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:03.398408] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:03.398430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:03.398443] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:03.398711] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:03.402610] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:03.402634] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:03.406514] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:06.036842] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:06.036868] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:06.036891] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:06.036904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:06.037156] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:06.040099] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:06.040120] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:06.043468] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:08.820160] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:08.820186] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:08.820209] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:08.820223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:08.820489] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:08.823592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:08.823618] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:08.826993] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:11.531675] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:11.531701] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:11.531724] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:11.531759] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:11.532027] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:11.540761] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:11.540795] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:11.544342] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:14.254845] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:14.254872] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:14.254897] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:14.254911] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:14.255173] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:14.258245] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:14.258269] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:05:14.270495] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file3: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:17.427195] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:17.427223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:17.427247] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:17.427261] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:17.427534] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file3: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:17.430679] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:17.430704] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file3 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:17.434592] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file4: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:20.136482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:20.136510] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:20.136534] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:20.136547] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:20.136819] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file4: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:20.139787] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:20.139823] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file4 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:20.142907] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file7: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:22.826287] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:22.826313] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:22.826335] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:22.826348] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:22.826654] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file7: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:22.829577] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:22.829598] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file7 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:22.832859] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file9: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:25.555481] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:25.555506] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:25.555529] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:25.555542] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:25.555807] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file9: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:25.558779] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:25.558800] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file9 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:25.571043] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-11 22:05:25.576225] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-11 22:05:25.588947] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:28.280132] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:28.280158] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:28.280181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:28.280194] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:28.280458] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:28.283465] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:28.283487] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:28.286900] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:31.228011] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:31.228037] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:31.228060] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:31.228073] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:31.228321] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:31.235906] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:31.235930] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:31.239463] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:33.868689] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:33.868715] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:33.868757] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:33.868774] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:33.869022] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:33.876591] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:33.876612] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:33.899010] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-11 22:05:33.921160] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-11 22:05:33.945003] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:36.423172] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:36.423198] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:36.423220] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:36.423238] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:36.423526] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:36.427157] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:36.427179] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:36.430803] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:38.998375] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:38.998408] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:38.998433] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:38.998447] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:38.998693] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:39.001592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:39.001612] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:39.004932] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:41.895907] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:41.895933] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:41.895956] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:41.895968] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:41.896226] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:41.900557] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:41.900579] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:05:41.912595] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-11 22:05:41.922098] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-11 22:05:41.934337] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:44.633955] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:44.633982] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:44.634007] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:44.634029] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:44.637512] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:44.641571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:44.641601] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:44.645977] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:47.243064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:47.243090] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:47.243113] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:47.243125] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:47.246635] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:47.250499] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:47.250519] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:47.255013] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:49.849070] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:49.849096] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:49.849118] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:49.849131] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:49.849897] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:49.857510] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:49.857533] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:49.864434] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:52.434107] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:52.434136] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:05:52.434160] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:52.434183] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:05:52.434459] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:05:52.442429] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:52.442456] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:05:52.451666] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:55.247204] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:55.247243] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:55.247289] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:55.247307] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:55.247575] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:55.251386] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:55.251408] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:55.254976] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:58.315972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:58.316000] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:05:58.316024] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:05:58.316037] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:05:58.316318] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:05:58.319423] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:05:58.319448] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:05:58.323077] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:01.156870] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:01.156895] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:01.156918] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:01.156931] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:01.157197] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:01.160180] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:01.160202] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:01.171157] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-11 22:06:01.176407] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-11 22:06:01.187326] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:03.884952] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:03.884978] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:03.885000] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:03.885012] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:03.885261] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:03.888346] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:03.888368] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:03.891784] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:06.465185] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:06.465210] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:06.465232] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:06.465245] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:06.465497] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:06.468476] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:06.468498] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:06.471821] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:09.133660] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:09.133685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:09.133708] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:09.133728] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:09.133998] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:09.137134] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:09.137155] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-11 22:06:09.150832] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-11 22:06:09.155805] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-11 22:06:09.162711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file3: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:11.952428] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:11.952455] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:11.952477] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:11.952491] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:11.952760] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file3: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:11.955797] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:11.955819] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file3 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:11.959383] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file4: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:14.617348] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:14.617375] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:14.617398] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:14.617411] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:14.617671] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file4: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:14.620820] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:14.620842] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file4 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:14.624201] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file7: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:17.357807] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:17.357835] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:17.357868] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:17.357882] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:17.358137] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file7: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:17.361118] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:17.361140] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file7 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:17.368270] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file9: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:20.222037] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:20.222067] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-11 22:06:20.222090] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:20.222103] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-11 22:06:20.222370] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file9: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-11 22:06:20.225176] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:20.225198] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file9 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-11 22:06:20.230802] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file5: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:22.999380] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:22.999406] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:22.999428] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:22.999441] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:22.999693] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:23.002794] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:23.002816] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file5 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:23.006186] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file6: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:25.824391] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:25.824417] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:25.824451] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:25.824475] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:25.824730] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:25.827721] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:25.827760] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file6 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:25.831113] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file8: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:28.473273] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:28.473298] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:28.473321] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:28.473334] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:28.473596] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:28.476603] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:28.476625] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file8 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-11 22:06:28.480210] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:31.096126] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:31.096152] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:31.096175] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:31.096188] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:31.096441] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:31.100080] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:31.100102] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:31.103635] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:33.815078] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:33.815105] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:33.815128] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:33.815148] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:33.815421] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:33.821044] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:33.821067] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:33.824588] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:36.406969] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:36.406996] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-11 22:06:36.407019] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-11 22:06:36.407033] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-11 22:06:36.407302] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-11 22:06:36.410762] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-11 22:06:36.410787] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-11 22:06:36.425111] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 49, size: 4900000000, lookups: 115
>[2012-04-11 22:06:36.425465] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-12 20:26:00.009545] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-12 20:26:05.033233] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-12 20:26:05.036009] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-12 20:26:05.111797] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-12 20:26:05.111835] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5'
>[2012-04-12 20:26:05.111856] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:26:05.111875] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:26:05.112247] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:26:05.112486] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-12 20:26:05.112598] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-12 20:26:05.112789] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2
>[2012-04-12 20:26:05.143396] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-12 20:26:05.148170] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-12 20:26:05.154083] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-12 20:26:05.158680] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-12 20:26:05.163422] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-12 20:26:05.168174] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     option decommissioned-bricks master-replicate-2
> 73:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 74: end-volume
> 75: 
> 76: volume master-write-behind
> 77:     type performance/write-behind
> 78:     subvolumes master-dht
> 79: end-volume
> 80: 
> 81: volume master-read-ahead
> 82:     type performance/read-ahead
> 83:     subvolumes master-write-behind
> 84: end-volume
> 85: 
> 86: volume master-io-cache
> 87:     type performance/io-cache
> 88:     subvolumes master-read-ahead
> 89: end-volume
> 90: 
> 91: volume master-quick-read
> 92:     type performance/quick-read
> 93:     subvolumes master-io-cache
> 94: end-volume
> 95: 
> 96: volume master-md-cache
> 97:     type performance/md-cache
> 98:     subvolumes master-quick-read
> 99: end-volume
>100: 
>101: volume master
>102:     type debug/io-stats
>103:     option latency-measurement off
>104:     option count-fop-hits off
>105:     subvolumes master-md-cache
>106: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-12 20:26:05.174890] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-12 20:26:05.175057] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-12 20:26:05.175202] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-12 20:26:05.175362] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-12 20:26:05.175504] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24015 (from 0)
>[2012-04-12 20:26:05.175646] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24016 (from 0)
>[2012-04-12 20:26:05.175870] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-12 20:26:05.175929] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-12 20:26:05.175971] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-12 20:26:05.176009] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-12 20:26:05.176047] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-12 20:26:05.176085] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-12 20:26:09.066177] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-12 20:26:09.070917] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-12 20:26:09.075709] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-12 20:26:09.080431] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-12 20:26:09.085145] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-12 20:26:09.087896] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.088065] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.088173] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.088277] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.088404] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.091892] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-12 20:26:09.092069] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:26:09.110317] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-12 20:26:09.110352] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.110659] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-12 20:26:09.110727] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-12 20:26:09.110796] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.110887] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-04-12 20:26:09.110946] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-12 20:26:09.110969] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.112408] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-12 20:26:09.112442] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.113271] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-12 20:26:09.113332] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24015, attached to remote volume '/exportdir/master/d5'.
>[2012-04-12 20:26:09.113356] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.113607] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-12 20:26:09.113664] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-12 20:26:09.113702] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-12 20:26:09.113777] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24016, attached to remote volume '/exportdir/master/d6'.
>[2012-04-12 20:26:09.113805] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:26:09.114662] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-12 20:26:09.114724] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-12 20:26:09.114823] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-12 20:26:09.125140] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-12 20:26:09.128189] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-12 20:26:09.128329] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-12 20:26:09.145197] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-12 20:26:09.167863] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-12 20:26:09.195873] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-12 20:26:09.205499] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-12 20:26:09.260040] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:14.732047] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:14.747961] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:14.748018] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:14.748053] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:14.748539] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:14.753694] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:14.753730] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:14.763711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:18.904931] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:18.904972] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:18.905011] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:18.905034] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:18.905478] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:18.911571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:18.911607] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:18.916598] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:21.875703] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:21.875775] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:21.875822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:21.875846] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:21.876272] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:21.881852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:21.881889] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:21.893505] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-12 20:26:21.902717] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-12 20:26:22.147508] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:25.817787] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:25.817829] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:25.817886] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:25.817910] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:25.818335] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:25.827265] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:25.827303] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:25.835722] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:28.972723] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:28.972795] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:28.976710] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:28.976779] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:28.977218] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:28.982042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:28.982079] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:28.987120] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:32.054278] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:32.054320] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:32.054361] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:32.054384] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:32.054859] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:32.059815] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:32.059852] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:26:32.071489] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-12 20:26:32.096449] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-12 20:26:32.243951] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:35.640953] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:35.641005] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:35.641047] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:35.641069] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:35.641455] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:35.647809] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:35.647839] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:35.652280] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:38.708734] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:38.708804] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:38.708847] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:38.708871] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:38.709315] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:38.714252] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:38.714289] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:38.719349] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:41.906924] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:41.906963] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:41.907002] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:41.907025] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:41.907472] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:41.912422] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:41.912459] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:41.924123] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-12 20:26:41.942845] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-12 20:26:42.149156] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:45.238800] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:45.238841] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:45.238881] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:45.238904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:45.239333] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:45.249829] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:45.249868] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:45.255164] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:49.019833] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:49.019873] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:49.019913] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:49.019936] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:49.020382] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:49.025339] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:49.025376] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:49.030506] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:52.802267] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:52.802305] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:52.802345] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:52.802368] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:52.802841] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:52.811699] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:52.811767] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:52.834163] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-12 20:26:52.848951] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-12 20:26:52.919144] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:57.544461] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:57.544504] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:26:57.544543] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:26:57.544567] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:26:57.545028] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:26:57.556919] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:26:57.556957] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:26:57.566627] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:27:00.751037] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:00.751079] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:00.751119] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:00.751142] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:00.751587] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:00.756541] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:00.756578] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:27:00.761705] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:27:03.991763] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:03.991791] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:03.991814] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:03.991827] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:03.992079] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:03.994848] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:03.994870] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:27:04.001539] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-12 20:27:04.006724] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-12 20:27:04.046685] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:07.290509] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:07.290549] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:27:07.290589] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:07.290612] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:27:07.291070] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:27:07.296145] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:07.296182] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:07.301321] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:10.747532] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:10.747576] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:27:10.762482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:10.762518] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:27:10.762987] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:27:10.768026] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:10.768063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:10.773206] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:13.787114] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:13.787154] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:27:13.787193] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:13.787217] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:27:13.787659] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:27:13.792673] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:13.792709] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:27:13.809539] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:17.143891] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:17.143917] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:17.143941] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:17.143954] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:17.144207] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:17.147139] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:17.147161] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:17.150328] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:20.444851] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:20.444891] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:20.444931] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:20.444954] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:20.445400] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:20.450449] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:20.450485] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:20.455772] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:23.110333] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:23.110361] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:23.110397] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:23.110411] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:23.110656] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:23.113553] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:23.113575] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:23.116674] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:26.270671] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:26.270709] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:26.270781] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:26.270810] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:26.271240] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:26.276272] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:26.276307] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:26.288085] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-12 20:27:26.297215] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-12 20:27:26.365204] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:29.760181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:29.760221] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:29.760260] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:29.760283] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:29.760722] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:29.765719] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:29.765785] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:29.770926] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:32.965305] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:32.965347] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:32.965656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:32.965688] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:32.966146] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:32.970984] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:32.971036] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:32.976162] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:36.460009] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:36.460050] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:36.460090] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:36.460114] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:36.460555] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:36.465707] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:36.465773] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:36.477544] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-12 20:27:36.486646] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-12 20:27:36.544917] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:39.693023] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:39.693057] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:39.693095] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:39.693116] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:39.693480] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:39.696516] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:39.696537] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:39.699518] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:42.597229] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:42.597272] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:42.597312] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:42.597335] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:42.597809] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:42.604328] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:42.604364] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:42.609447] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:45.613252] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:45.613291] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:27:45.613332] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:27:45.613356] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:27:45.613825] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:27:45.618761] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:27:45.618799] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:27:45.671517] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 83
>[2012-04-12 20:27:45.812429] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-12 20:30:05.335288] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-12 20:30:10.360597] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-12 20:30:10.360677] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-12 20:30:10.365924] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-12 20:30:10.365958] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1'
>[2012-04-12 20:30:10.365980] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:30:10.365999] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:30:10.366019] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:30:10.366222] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-12 20:30:10.366355] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-12 20:30:10.373998] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-12 20:30:10.378887] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-12 20:30:10.383538] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-12 20:30:10.388724] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-replicate-0
> 38:     type cluster/replicate
> 39:     subvolumes master-client-0 master-client-1
> 40: end-volume
> 41: 
> 42: volume master-replicate-1
> 43:     type cluster/replicate
> 44:     subvolumes master-client-2 master-client-3
> 45: end-volume
> 46: 
> 47: volume master-dht
> 48:     type cluster/distribute
> 49:     subvolumes master-replicate-0 master-replicate-1
> 50: end-volume
> 51: 
> 52: volume master-write-behind
> 53:     type performance/write-behind
> 54:     subvolumes master-dht
> 55: end-volume
> 56: 
> 57: volume master-read-ahead
> 58:     type performance/read-ahead
> 59:     subvolumes master-write-behind
> 60: end-volume
> 61: 
> 62: volume master-io-cache
> 63:     type performance/io-cache
> 64:     subvolumes master-read-ahead
> 65: end-volume
> 66: 
> 67: volume master-quick-read
> 68:     type performance/quick-read
> 69:     subvolumes master-io-cache
> 70: end-volume
> 71: 
> 72: volume master-md-cache
> 73:     type performance/md-cache
> 74:     subvolumes master-quick-read
> 75: end-volume
> 76: 
> 77: volume master
> 78:     type debug/io-stats
> 79:     option latency-measurement off
> 80:     option count-fop-hits off
> 81:     subvolumes master-md-cache
> 82: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-12 20:30:10.395605] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-12 20:30:10.395800] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-12 20:30:10.395952] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-12 20:30:10.396095] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-12 20:30:10.396250] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-12 20:30:10.396324] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-12 20:30:10.396369] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-12 20:30:10.396408] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-12 20:30:14.357429] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-12 20:30:14.357631] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:30:14.358081] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-12 20:30:14.358132] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:30:14.358648] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-04-12 20:30:14.358878] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-12 20:30:14.363823] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-12 20:30:14.363990] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:30:14.364347] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-12 20:30:14.364396] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:30:14.365385] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-12 20:30:14.370122] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-12 20:30:14.370283] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:30:14.370650] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-12 20:30:14.370683] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:30:14.370808] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-12 20:30:14.370997] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-12 20:30:14.375687] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-12 20:30:14.375879] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:30:14.376264] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-12 20:30:14.376303] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:30:14.377183] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-12 20:30:14.396696] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-12 20:30:14.399077] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-12 20:30:14.399343] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-12 20:30:14.405131] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-12 20:30:14.418664] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-12 20:30:14.425138] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-12 20:30:14.454548] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-12 20:30:14.461089] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-12 20:30:14.490450] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-12 20:30:14.497065] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-12 20:30:14.526405] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-12 20:30:14.532919] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-12 20:30:14.561986] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-12 20:30:14.568239] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-12 20:30:14.597112] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-12 20:30:14.603391] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-12 20:30:14.631910] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-12 20:30:14.638100] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-12 20:30:14.666473] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-12 20:30:14.672683] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-12 20:30:14.700076] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 80
>[2012-04-12 20:30:14.717664] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-12 20:39:42.127518] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-12 20:39:47.129680] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-12 20:39:47.130877] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-12 20:39:47.164946] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-12 20:39:47.164987] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1'
>[2012-04-12 20:39:47.165008] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:39:47.165028] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:39:47.165047] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:39:47.165276] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-12 20:39:47.165389] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-12 20:39:47.229102] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-12 20:39:47.233982] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-12 20:39:47.238605] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-12 20:39:47.243102] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-12 20:39:47.247606] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-12 20:39:47.252138] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 73: end-volume
> 74: 
> 75: volume master-write-behind
> 76:     type performance/write-behind
> 77:     subvolumes master-dht
> 78: end-volume
> 79: 
> 80: volume master-read-ahead
> 81:     type performance/read-ahead
> 82:     subvolumes master-write-behind
> 83: end-volume
> 84: 
> 85: volume master-io-cache
> 86:     type performance/io-cache
> 87:     subvolumes master-read-ahead
> 88: end-volume
> 89: 
> 90: volume master-quick-read
> 91:     type performance/quick-read
> 92:     subvolumes master-io-cache
> 93: end-volume
> 94: 
> 95: volume master-md-cache
> 96:     type performance/md-cache
> 97:     subvolumes master-quick-read
> 98: end-volume
> 99: 
>100: volume master
>101:     type debug/io-stats
>102:     option latency-measurement off
>103:     option count-fop-hits off
>104:     subvolumes master-md-cache
>105: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-12 20:39:47.258862] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-12 20:39:47.259033] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-12 20:39:47.259180] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-12 20:39:47.259325] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-12 20:39:47.259485] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24017 (from 0)
>[2012-04-12 20:39:47.259631] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24018 (from 0)
>[2012-04-12 20:39:47.259819] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-12 20:39:47.259879] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-12 20:39:47.259931] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-12 20:39:47.259973] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-12 20:39:47.260012] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-12 20:39:47.260049] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-12 20:39:51.145395] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-12 20:39:51.145569] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.146481] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-12 20:39:51.146515] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.146833] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online.
>[2012-04-12 20:39:51.147027] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-12 20:39:51.151374] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-12 20:39:51.151532] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.151915] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-12 20:39:51.151950] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.152789] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-12 20:39:51.157445] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-12 20:39:51.157600] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.157986] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-12 20:39:51.158021] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.158106] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online.
>[2012-04-12 20:39:51.158306] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-12 20:39:51.162978] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-12 20:39:51.163134] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.163500] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-12 20:39:51.163533] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.164306] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-12 20:39:51.169053] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-12 20:39:51.169209] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.169572] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24017, attached to remote volume '/exportdir/master/d5'.
>[2012-04-12 20:39:51.169615] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.169698] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-12 20:39:51.169916] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-12 20:39:51.174607] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-12 20:39:51.174798] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:39:51.175160] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24018, attached to remote volume '/exportdir/master/d6'.
>[2012-04-12 20:39:51.175193] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:39:51.176053] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-12 20:39:51.188166] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-12 20:39:51.191427] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-12 20:39:51.191575] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-12 20:39:51.191856] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-12 20:39:51.200429] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-12 20:39:51.219418] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-12 20:39:51.229238] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-12 20:39:51.250526] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:39:54.692654] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:39:54.692691] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:39:54.692728] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:39:54.692783] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:39:54.693188] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:39:54.697887] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:39:54.697920] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:39:54.701005] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:39:58.382487] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:39:58.382526] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:39:58.382567] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:39:58.382600] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:39:58.383071] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:39:58.388051] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:39:58.388099] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:39:58.393127] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:01.783927] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:01.783967] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:01.784006] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:01.784029] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:01.784473] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:01.789616] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:01.789651] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:01.807316] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-12 20:40:01.816388] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-12 20:40:01.835361] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:06.179182] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:06.179223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:06.179262] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:06.179285] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:06.179727] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:06.184708] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:06.184773] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:06.190585] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:09.489136] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:09.489176] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:09.489226] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:09.489250] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:09.489696] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:09.505899] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:09.505937] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:09.515191] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:12.479103] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:12.479143] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:12.479183] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:12.479206] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:12.479649] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:12.484712] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:12.484777] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:12.509271] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-12 20:40:12.518420] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-12 20:40:12.544716] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:15.507602] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:15.507644] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:15.507684] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:15.507708] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:15.508168] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:15.513078] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:15.513115] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:15.518166] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:18.478527] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:18.478584] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:18.478627] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:18.478651] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:18.479119] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:18.483972] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:18.484008] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:18.489147] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:21.598546] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:21.598582] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:21.598619] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:21.598640] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:21.599074] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:21.603721] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:21.603782] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:21.619706] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-12 20:40:21.628273] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-12 20:40:21.652842] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:24.576064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:24.576090] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:24.576114] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:24.576127] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:24.576408] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:24.579350] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:24.579371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:24.582465] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:27.871951] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:27.871991] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:27.872031] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:27.872055] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:27.872515] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:27.882878] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:27.882916] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:27.892342] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:31.036198] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:31.036223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:31.036246] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:31.036259] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:31.036522] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:31.039417] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:31.039443] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:31.049110] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-12 20:40:31.054337] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-12 20:40:31.069178] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:34.122825] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:34.122850] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:34.122872] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:34.122885] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:34.123134] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:34.126090] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:34.126110] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:34.129155] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:37.073972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:37.074012] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:37.074052] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:37.074075] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:37.074519] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:37.079472] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:37.079508] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:37.084525] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:40.088485] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:40.088539] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:40.088582] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:40.088606] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:40.089072] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:40.097388] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:40.097427] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-1 to master-replicate-2
>[2012-04-12 20:40:40.116789] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-12 20:40:40.125993] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-12 20:40:40.140520] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:43.258311] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:43.258350] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:43.258389] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:43.258412] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:43.258888] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:43.263981] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:43.264027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:43.269149] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:46.648289] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:46.648331] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:46.648372] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:46.648395] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:46.648894] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:46.653884] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:46.653935] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:46.659059] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:50.266066] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:50.266109] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:50.266150] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:50.266173] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:50.266618] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:50.271592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:50.271629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:50.276773] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:53.374956] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 69
>[2012-04-12 20:40:53.899955] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:53.899994] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:40:53.900035] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:53.900058] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:40:53.900486] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:40:53.909171] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:53.909222] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:40:53.927456] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-12 20:40:56.412977] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 73
>[2012-04-12 20:40:57.030706] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:57.030775] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:40:57.030822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:40:57.030846] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:40:57.031277] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:40:57.036334] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:40:57.036370] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-12 20:40:57.041446] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-12 20:41:00.166482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:00.166508] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:41:00.166533] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:00.166546] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:41:00.166819] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:41:00.169701] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:00.169722] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-12 20:41:00.172805] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-0
>[2012-04-12 20:41:03.137282] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:03.137308] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory
>[2012-04-12 20:41:03.137331] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:03.137344] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory
>[2012-04-12 20:41:03.137597] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory)
>[2012-04-12 20:41:03.140354] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:03.140375] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-0
>[2012-04-12 20:41:03.155087] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-12 20:41:03.161919] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-12 20:41:03.173531] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:06.268522] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:06.268561] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:06.268601] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:06.268624] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:06.269085] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:06.274025] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:06.274061] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:06.279236] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:09.689273] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:09.689313] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:09.689353] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:09.689376] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:09.689850] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:09.694786] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:09.694822] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:09.699869] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:12.856365] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:12.856404] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:12.856443] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:12.856465] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:12.856927] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:12.861916] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:12.861952] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:12.886852] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-12 20:41:12.896422] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-12 20:41:12.915913] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:16.078884] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:16.078926] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:16.078967] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:16.078990] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:16.079417] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:16.084419] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:16.084470] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:16.089568] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:19.074657] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:19.074697] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:19.074767] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:19.074798] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:19.075228] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:19.080307] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:19.080344] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:19.085441] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:22.389098] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:22.389140] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:41:22.389181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:41:22.389204] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:41:22.389666] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:41:22.394801] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:41:22.394838] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-0 to master-replicate-2
>[2012-04-12 20:41:22.418656] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 105
>[2012-04-12 20:41:22.419154] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-12 20:44:54.256571] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-12 20:44:59.258637] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-12 20:44:59.261066] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-12 20:44:59.358724] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942'
>[2012-04-12 20:44:59.358803] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5'
>[2012-04-12 20:44:59.358827] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:44:59.358846] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:44:59.358865] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-12 20:44:59.359102] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-12 20:44:59.359214] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-12 20:44:59.359392] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2
>[2012-04-12 20:44:59.423140] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-12 20:44:59.428137] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-12 20:44:59.432870] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-12 20:44:59.437415] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>[2012-04-12 20:44:59.441980] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport
>[2012-04-12 20:44:59.446561] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
>  7:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 16:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 25:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 34:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 35: end-volume
> 36: 
> 37: volume master-client-4
> 38:     type protocol/client
> 39:     option remote-host 172.17.251.56
> 40:     option remote-subvolume /exportdir/master/d5
> 41:     option transport-type tcp
> 42:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 43:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 44: end-volume
> 45: 
> 46: volume master-client-5
> 47:     type protocol/client
> 48:     option remote-host 172.17.251.56
> 49:     option remote-subvolume /exportdir/master/d6
> 50:     option transport-type tcp
> 51:     option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb
> 52:     option password 778680bb-2fb7-46a7-961e-5093d2ade291
> 53: end-volume
> 54: 
> 55: volume master-replicate-0
> 56:     type cluster/replicate
> 57:     subvolumes master-client-0 master-client-1
> 58: end-volume
> 59: 
> 60: volume master-replicate-1
> 61:     type cluster/replicate
> 62:     subvolumes master-client-2 master-client-3
> 63: end-volume
> 64: 
> 65: volume master-replicate-2
> 66:     type cluster/replicate
> 67:     subvolumes master-client-4 master-client-5
> 68: end-volume
> 69: 
> 70: volume master-dht
> 71:     type cluster/distribute
> 72:     option decommissioned-bricks master-replicate-2
> 73:     subvolumes master-replicate-0 master-replicate-1 master-replicate-2
> 74: end-volume
> 75: 
> 76: volume master-write-behind
> 77:     type performance/write-behind
> 78:     subvolumes master-dht
> 79: end-volume
> 80: 
> 81: volume master-read-ahead
> 82:     type performance/read-ahead
> 83:     subvolumes master-write-behind
> 84: end-volume
> 85: 
> 86: volume master-io-cache
> 87:     type performance/io-cache
> 88:     subvolumes master-read-ahead
> 89: end-volume
> 90: 
> 91: volume master-quick-read
> 92:     type performance/quick-read
> 93:     subvolumes master-io-cache
> 94: end-volume
> 95: 
> 96: volume master-md-cache
> 97:     type performance/md-cache
> 98:     subvolumes master-quick-read
> 99: end-volume
>100: 
>101: volume master
>102:     type debug/io-stats
>103:     option latency-measurement off
>104:     option count-fop-hits off
>105:     subvolumes master-md-cache
>106: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-12 20:44:59.453375] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-12 20:44:59.453540] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-12 20:44:59.453686] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-12 20:44:59.453864] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-12 20:44:59.454009] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24017 (from 0)
>[2012-04-12 20:44:59.454150] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24018 (from 0)
>[2012-04-12 20:44:59.454320] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-12 20:44:59.454377] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-12 20:44:59.454419] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-12 20:44:59.454458] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-12 20:44:59.454560] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer
>[2012-04-12 20:44:59.454606] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer
>[2012-04-12 20:45:03.275643] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-12 20:45:03.275878] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.280834] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-12 20:45:03.280999] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.286140] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-12 20:45:03.286305] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.291209] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-12 20:45:03.291389] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.295504] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-12 20:45:03.295540] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.295764] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-3' came back up; going online.
>[2012-04-12 20:45:03.295836] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-12 20:45:03.295861] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.296053] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-12 20:45:03.296084] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.296169] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-1' came back up; going online.
>[2012-04-12 20:45:03.296228] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-12 20:45:03.296253] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.297806] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-12 20:45:03.297859] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-12 20:45:03.297897] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-12 20:45:03.297932] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-12 20:45:03.299777] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer
>[2012-04-12 20:45:03.299941] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.300306] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24017, attached to remote volume '/exportdir/master/d5'.
>[2012-04-12 20:45:03.300351] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.300444] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online.
>[2012-04-12 20:45:03.300628] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1
>[2012-04-12 20:45:03.305411] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer
>[2012-04-12 20:45:03.305576] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 20:45:03.305975] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24018, attached to remote volume '/exportdir/master/d6'.
>[2012-04-12 20:45:03.306008] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 20:45:03.306863] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1
>[2012-04-12 20:45:03.334275] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode
>[2012-04-12 20:45:03.337545] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode
>[2012-04-12 20:45:03.337689] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode
>[2012-04-12 20:45:03.337973] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-12 20:45:03.346607] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-12 20:45:03.365590] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-12 20:45:03.375277] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-12 20:45:03.402009] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:07.453222] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:07.453274] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:07.453335] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:07.453387] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:07.453956] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:07.458949] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:07.458984] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:07.462322] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:11.692414] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:11.692453] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:11.692491] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:11.692514] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:11.692981] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:11.697950] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:11.697986] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:11.702949] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:14.788337] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:14.788389] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:14.788430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:14.788453] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:14.788915] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:14.793890] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:14.793926] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:14.805547] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-12 20:45:14.814636] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-12 20:45:14.840993] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:17.820679] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:17.820718] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:17.820790] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:17.820817] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:17.821242] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:17.826203] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:17.826239] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:17.835407] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:20.821801] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:20.821841] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:20.821880] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:20.821915] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:20.822361] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:20.827335] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:20.827371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:20.836908] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:23.823500] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:23.823538] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:23.823577] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:23.823600] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:23.824056] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:23.829009] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:23.829045] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:45:23.840710] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-12 20:45:23.849813] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-12 20:45:23.876431] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:26.968516] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:26.968556] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:26.977098] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:26.977150] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:26.977655] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:26.993672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:26.993728] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:27.010275] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:29.839283] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:29.839323] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:29.839395] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:29.839421] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:29.839879] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:29.844812] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:29.844849] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:29.854113] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:32.827691] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:32.827730] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:32.827808] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:32.827831] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:32.828258] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:32.833209] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:32.833244] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:32.844954] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-12 20:45:32.854023] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-12 20:45:32.880758] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:35.921384] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:35.921423] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:35.921462] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:35.921492] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:35.922425] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:35.938923] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:35.938966] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:35.955258] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:39.083448] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:39.083506] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:39.083548] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:39.083571] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:39.084031] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:39.089056] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:39.089092] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:39.098765] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:42.395767] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:42.395808] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:42.395850] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:42.395872] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:42.396315] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:42.404852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:42.404890] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:42.427159] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-12 20:45:42.439253] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-12 20:45:42.465991] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:45.686835] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:45.686861] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:45.686883] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:45.686896] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:45.687149] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:45.690190] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:45.690211] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:45.696103] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:48.861972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:48.862014] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:48.862054] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:48.862076] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:48.862517] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:48.870169] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:48.870205] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:48.879810] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:52.309822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:52.309877] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:45:52.309926] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:52.309950] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:45:52.310397] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:45:52.315485] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:52.315521] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-2 to master-replicate-1
>[2012-04-12 20:45:52.327398] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-12 20:45:52.336554] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-12 20:45:52.351134] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:45:55.508373] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:55.508414] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:45:55.508652] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:55.508685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:45:55.509147] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:45:55.533135] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:55.533172] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:45:55.549380] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:45:56.445287] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 52
>[2012-04-12 20:45:58.425216] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 52
>[2012-04-12 20:45:58.973727] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:58.973797] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:45:58.973839] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:45:58.973862] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:45:58.974290] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:45:58.979342] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:45:58.979379] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:45:58.989100] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-0 to master-replicate-1
>[2012-04-12 20:46:00.328311] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 53
>[2012-04-12 20:46:01.902410] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 53
>[2012-04-12 20:46:02.223591] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:02.223628] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory
>[2012-04-12 20:46:02.223666] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:02.223689] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory
>[2012-04-12 20:46:02.224144] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-0 (No such file or directory)
>[2012-04-12 20:46:02.233249] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:02.233300] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-0 to master-replicate-1
>[2012-04-12 20:46:02.269474] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:05.275793] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:05.275836] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:05.276526] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:05.276561] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:05.277058] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:05.284349] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:05.284387] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:05.301663] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:08.238831] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:08.238870] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:08.238909] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:08.238931] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:08.239373] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:08.244399] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:08.244435] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:08.253999] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:09.826882] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 62
>[2012-04-12 20:46:11.506982] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:11.507020] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:11.507059] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:11.507081] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:11.507524] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:11.514084] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:11.514122] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:11.524271] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:11.948469] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 63
>[2012-04-12 20:46:13.699874] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 63
>[2012-04-12 20:46:14.715562] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:14.715586] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:14.715618] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:14.715632] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:14.715902] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:14.718817] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:14.718839] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:14.725571] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-12 20:46:14.730794] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-12 20:46:14.756481] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:15.669232] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 71
>[2012-04-12 20:46:17.463769] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 71
>[2012-04-12 20:46:18.132200] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:18.132227] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:18.132253] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:18.132269] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:18.132572] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:18.135936] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:18.135960] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:18.142251] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:19.093861] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 72
>[2012-04-12 20:46:20.907469] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 72
>[2012-04-12 20:46:21.495123] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:21.495160] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:21.495199] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:21.495221] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:21.495662] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:21.500627] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:21.500675] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:21.510001] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:24.551678] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:24.551717] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:24.551789] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:24.551817] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:24.552244] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:24.557261] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:24.557311] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:24.569065] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-12 20:46:24.578248] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-12 20:46:24.605120] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:27.615048] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:27.615074] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:27.615097] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:27.615110] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:27.615380] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:27.618312] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:27.618334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:27.624909] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:30.632502] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:30.632529] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:30.632552] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:30.632565] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:30.632835] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:30.635608] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:30.635629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:30.644212] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:34.118193] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:34.118232] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory
>[2012-04-12 20:46:34.118271] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null)
>[2012-04-12 20:46:34.118294] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory
>[2012-04-12 20:46:34.118767] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory)
>[2012-04-12 20:46:34.123812] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 20:46:34.123848] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-0
>[2012-04-12 20:46:34.135246] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 83
>[2012-04-12 20:46:34.148565] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-12 23:19:42.469273] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-12 23:19:47.482678] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-12 23:19:47.482790] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-12 23:19:47.485636] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '9d5940f1-a0b4-4fee-910c-d8cba200af43'
>[2012-04-12 23:19:47.485682] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1'
>[2012-04-12 23:19:47.485717] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-12 23:19:47.487319] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-12 23:19:47.487366] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-12 23:19:47.487695] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-12 23:19:47.487918] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-12 23:19:47.493574] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-12 23:19:47.498327] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-12 23:19:47.503086] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-12 23:19:47.507707] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
>  7:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 16:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 25:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 34:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 35: end-volume
> 36: 
> 37: volume master-dht
> 38:     type cluster/distribute
> 39:     subvolumes master-client-0 master-client-1 master-client-2 master-client-3
> 40: end-volume
> 41: 
> 42: volume master-write-behind
> 43:     type performance/write-behind
> 44:     subvolumes master-dht
> 45: end-volume
> 46: 
> 47: volume master-read-ahead
> 48:     type performance/read-ahead
> 49:     subvolumes master-write-behind
> 50: end-volume
> 51: 
> 52: volume master-io-cache
> 53:     type performance/io-cache
> 54:     subvolumes master-read-ahead
> 55: end-volume
> 56: 
> 57: volume master-quick-read
> 58:     type performance/quick-read
> 59:     subvolumes master-io-cache
> 60: end-volume
> 61: 
> 62: volume master-md-cache
> 63:     type performance/md-cache
> 64:     subvolumes master-quick-read
> 65: end-volume
> 66: 
> 67: volume master
> 68:     type debug/io-stats
> 69:     option latency-measurement off
> 70:     option count-fop-hits off
> 71:     subvolumes master-md-cache
> 72: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-12 23:19:47.518328] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-12 23:19:47.518505] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-12 23:19:47.518653] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-12 23:19:47.518834] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-12 23:19:47.518988] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-12 23:19:47.519044] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-12 23:19:47.519086] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-12 23:19:47.519124] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-12 23:19:51.499621] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-12 23:19:51.499857] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 23:19:51.500290] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-12 23:19:51.500332] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 23:19:51.501078] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-12 23:19:51.505923] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-12 23:19:51.506087] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 23:19:51.506444] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-12 23:19:51.506492] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 23:19:51.506987] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-12 23:19:51.511801] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-12 23:19:51.511965] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 23:19:51.512320] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-12 23:19:51.512353] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 23:19:51.512851] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-12 23:19:51.517719] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-12 23:19:51.517917] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-12 23:19:51.518275] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-12 23:19:51.518308] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-12 23:19:51.518883] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-12 23:19:51.531871] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-12 23:19:51.535557] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-12 23:19:51.550146] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-12 23:19:51.554874] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-12 23:19:51.579011] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:19:54.110931] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:19:54.110989] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:19:54.113417] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:19:54.113454] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:19:54.115399] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:19:56.426239] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:19:56.426322] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:19:56.429274] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:19:56.429332] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:19:56.440396] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:19:58.288080] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:19:58.288169] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:19:58.290496] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:19:58.290530] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:19:58.300834] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file7: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:00.411185] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:00.411225] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file7: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:00.412766] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:00.412789] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file7 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:00.421956] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-12 23:20:00.424450] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-12 23:20:00.431001] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file5: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:02.079188] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:02.079246] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file5: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:02.081704] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:02.081772] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file5 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:02.085254] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file6: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:03.523787] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:03.523848] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file6: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:03.526273] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:03.526310] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file6 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:03.529834] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file8: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:04.975146] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:04.975216] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file8: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:04.977636] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:04.977672] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file8 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:04.984195] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file7: attempting to move from master-client-1 to master-client-2
>[2012-04-12 23:20:06.855169] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:06.855207] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file7: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:06.856666] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:06.856688] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file7 from subvolume master-client-1 to master-client-2
>[2012-04-12 23:20:06.865276] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-12 23:20:06.867592] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-12 23:20:06.873876] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:08.819055] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:08.819114] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:08.821627] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:08.821664] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:08.825200] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:10.745116] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:10.745154] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:10.746461] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:10.746482] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:10.748548] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-client-0 to master-client-3
>[2012-04-12 23:20:12.144115] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:12.144151] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:12.145634] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:12.145655] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-client-0 to master-client-3
>[2012-04-12 23:20:12.150173] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file7: attempting to move from master-client-1 to master-client-2
>[2012-04-12 23:20:13.578887] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:13.578946] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file7: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:13.582820] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:13.582857] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file7 from subvolume master-client-1 to master-client-2
>[2012-04-12 23:20:13.597882] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-12 23:20:13.602270] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-12 23:20:13.613676] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file7: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:15.395980] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:15.396040] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file7: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:15.398445] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:15.398482] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file7 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:15.406829] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:17.159310] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:17.159384] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:17.161804] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:17.161841] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:17.165312] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:18.744191] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:18.744230] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:18.745659] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:18.745680] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:18.747822] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:19.862278] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:19.862316] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:19.863754] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:19.863776] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:19.874613] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-12 23:20:19.876936] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-12 23:20:19.882242] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file7: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:21.160237] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:21.160285] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file7: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:21.162585] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:21.162614] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file7 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:21.169903] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:21.391109] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 64
>[2012-04-12 23:20:22.298206] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:22.298245] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:22.302059] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:22.302081] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:22.304192] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:23.325472] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 65
>[2012-04-12 23:20:23.370436] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:23.370470] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:23.371956] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:23.371978] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:23.374165] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:24.607866] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:24.607924] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:24.610359] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:24.610395] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:24.624842] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-12 23:20:24.629066] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-12 23:20:24.638297] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:25.533425] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 73
>[2012-04-12 23:20:26.087571] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:26.087608] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:26.089058] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:26.089079] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:26.093947] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file5: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:28.044091] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:28.044148] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file5: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:28.046564] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:28.046600] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file5 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:28.050177] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file6: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:29.368453] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:29.368503] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file6: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:29.370042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:29.370063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file6 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:29.372173] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file8: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:30.496811] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:30.496854] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file8: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:30.498613] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:30.498638] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file8 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:30.508833] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-12 23:20:30.511653] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-12 23:20:30.518878] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file5: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:31.700983] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:31.701027] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file5: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:31.702796] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:31.702821] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file5 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:31.706884] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file6: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:33.218043] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:33.218081] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file6: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:33.219588] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:33.219610] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file6 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:33.221816] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file8: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:34.387094] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:34.387130] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file8: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:34.388782] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:34.388804] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file8 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:34.392991] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file7: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:35.817452] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:35.817491] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file7: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:35.819005] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:35.819027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file7 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:35.827619] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-12 23:20:35.829973] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-12 23:20:35.835316] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file7: attempting to move from master-client-0 to master-client-2
>[2012-04-12 23:20:37.287588] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-12 23:20:37.287625] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file7: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-12 23:20:37.289100] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file7: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:37.289122] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file7 from subvolume master-client-0 to master-client-2
>[2012-04-12 23:20:37.294014] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file5: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:38.553869] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:38.553938] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file5: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:38.556369] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file5: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:38.556406] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file5 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:38.559940] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file6: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:40.126751] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:40.126792] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file6: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:40.128197] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file6: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:40.128218] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file6 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:40.130512] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file8: attempting to move from master-client-1 to master-client-3
>[2012-04-12 23:20:41.256249] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-12 23:20:41.256293] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file8: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-12 23:20:41.257825] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file8: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-12 23:20:41.257847] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file8 from subvolume master-client-1 to master-client-3
>[2012-04-12 23:20:41.267841] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 32, size: 3200000000, lookups: 112
>[2012-04-12 23:20:41.268153] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
>[2012-04-13 00:06:20.438274] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git
>[2012-04-13 00:06:25.450213] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8
>[2012-04-13 00:06:25.450948] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
>[2012-04-13 00:06:25.530734] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '9d5940f1-a0b4-4fee-910c-d8cba200af43'
>[2012-04-13 00:06:25.530823] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5'
>[2012-04-13 00:06:25.530849] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes'
>[2012-04-13 00:06:25.530868] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes'
>[2012-04-13 00:06:25.530887] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes'
>[2012-04-13 00:06:25.531075] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824
>[2012-04-13 00:06:25.531189] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824
>[2012-04-13 00:06:25.531354] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-client-0
>[2012-04-13 00:06:25.531413] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-client-1
>[2012-04-13 00:06:25.536078] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport
>[2012-04-13 00:06:25.540894] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport
>[2012-04-13 00:06:25.545450] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport
>[2012-04-13 00:06:25.550064] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume master-client-0
>  2:     type protocol/client
>  3:     option remote-host 172.17.251.56
>  4:     option remote-subvolume /exportdir/master/d1
>  5:     option transport-type tcp
>  6:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
>  7:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
>  8: end-volume
>  9: 
> 10: volume master-client-1
> 11:     type protocol/client
> 12:     option remote-host 172.17.251.56
> 13:     option remote-subvolume /exportdir/master/d2
> 14:     option transport-type tcp
> 15:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 16:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 17: end-volume
> 18: 
> 19: volume master-client-2
> 20:     type protocol/client
> 21:     option remote-host 172.17.251.56
> 22:     option remote-subvolume /exportdir/master/d3
> 23:     option transport-type tcp
> 24:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 25:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 26: end-volume
> 27: 
> 28: volume master-client-3
> 29:     type protocol/client
> 30:     option remote-host 172.17.251.56
> 31:     option remote-subvolume /exportdir/master/d4
> 32:     option transport-type tcp
> 33:     option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb
> 34:     option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6
> 35: end-volume
> 36: 
> 37: volume master-dht
> 38:     type cluster/distribute
> 39:     option decommissioned-bricks master-client-0,master-client-1
> 40:     subvolumes master-client-0 master-client-1 master-client-2 master-client-3
> 41: end-volume
> 42: 
> 43: volume master-write-behind
> 44:     type performance/write-behind
> 45:     subvolumes master-dht
> 46: end-volume
> 47: 
> 48: volume master-read-ahead
> 49:     type performance/read-ahead
> 50:     subvolumes master-write-behind
> 51: end-volume
> 52: 
> 53: volume master-io-cache
> 54:     type performance/io-cache
> 55:     subvolumes master-read-ahead
> 56: end-volume
> 57: 
> 58: volume master-quick-read
> 59:     type performance/quick-read
> 60:     subvolumes master-io-cache
> 61: end-volume
> 62: 
> 63: volume master-md-cache
> 64:     type performance/md-cache
> 65:     subvolumes master-quick-read
> 66: end-volume
> 67: 
> 68: volume master
> 69:     type debug/io-stats
> 70:     option latency-measurement off
> 71:     option count-fop-hits off
> 72:     subvolumes master-md-cache
> 73: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-04-13 00:06:25.556121] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0)
>[2012-04-13 00:06:25.556284] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0)
>[2012-04-13 00:06:25.556448] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0)
>[2012-04-13 00:06:25.556592] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0)
>[2012-04-13 00:06:25.556795] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer
>[2012-04-13 00:06:25.556884] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer
>[2012-04-13 00:06:25.556933] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer
>[2012-04-13 00:06:25.556985] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer
>[2012-04-13 00:06:29.468618] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer
>[2012-04-13 00:06:29.471047] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-13 00:06:29.474010] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer
>[2012-04-13 00:06:29.474178] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-13 00:06:29.479491] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer
>[2012-04-13 00:06:29.479662] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-13 00:06:29.484827] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer
>[2012-04-13 00:06:29.484998] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330)
>[2012-04-13 00:06:29.489363] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'.
>[2012-04-13 00:06:29.489412] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-13 00:06:29.489733] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'.
>[2012-04-13 00:06:29.489798] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-13 00:06:29.489929] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'.
>[2012-04-13 00:06:29.489958] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-13 00:06:29.490082] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'.
>[2012-04-13 00:06:29.490111] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-04-13 00:06:29.491776] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1
>[2012-04-13 00:06:29.491831] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1
>[2012-04-13 00:06:29.491868] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1
>[2012-04-13 00:06:29.491902] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1
>[2012-04-13 00:06:29.505006] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /
>[2012-04-13 00:06:29.508661] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /
>[2012-04-13 00:06:29.522890] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1
>[2012-04-13 00:06:29.527382] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1
>[2012-04-13 00:06:29.533610] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:06:31.701142] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:31.701202] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:31.703611] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:31.703647] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:06:31.707593] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:06:33.312177] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:33.312215] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:33.313651] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:33.313672] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:06:33.317229] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:06:34.670151] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:34.670210] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:34.672628] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:34.672665] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:06:34.679757] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file3: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:06:36.092096] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:36.092153] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file3: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:36.094982] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:36.095019] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file3 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:06:36.101007] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file4: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:06:37.445182] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:37.445220] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file4: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:37.448127] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:37.448149] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file4 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:06:37.451829] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file9: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:06:38.767356] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:38.767414] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file9: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:38.769857] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:38.769894] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file9 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:06:38.790623] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2
>[2012-04-13 00:06:38.794974] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2
>[2012-04-13 00:06:38.802711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:40.269666] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 17
>[2012-04-13 00:06:40.673495] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:40.673553] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:40.675990] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:40.676027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:40.679683] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:42.032099] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 18
>[2012-04-13 00:06:42.218490] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:42.218527] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:42.220010] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:42.220033] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:42.222605] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:43.544401] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:43.544460] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:43.546949] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:43.546987] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:43.551714] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file3: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:43.560925] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 900000000, lookups: 20
>[2012-04-13 00:06:44.842580] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 900000000, lookups: 20
>[2012-04-13 00:06:45.677568] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:45.677625] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file3: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:45.679966] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:45.680011] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file3 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:45.684084] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file4: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:46.691963] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 10, size: 1000000000, lookups: 21
>[2012-04-13 00:06:47.222276] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 10, size: 1000000000, lookups: 21
>[2012-04-13 00:06:47.812602] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:47.812658] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file4: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:47.815045] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:47.815081] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file4 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:47.818713] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file9: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:49.190537] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 11, size: 1100000000, lookups: 22
>[2012-04-13 00:06:49.373156] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:49.373191] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file9: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:49.374608] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:49.374629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file9 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:49.385681] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3
>[2012-04-13 00:06:49.388026] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3
>[2012-04-13 00:06:49.392249] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:50.714130] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 12, size: 1200000000, lookups: 33
>[2012-04-13 00:06:50.923195] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:50.923236] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:50.924680] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:50.924701] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:50.927077] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:52.234214] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 13, size: 1300000000, lookups: 34
>[2012-04-13 00:06:52.468341] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:52.468388] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:52.469828] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:52.469849] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:52.472201] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-client-0 to master-client-3
>[2012-04-13 00:06:53.811692] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 1400000000, lookups: 35
>[2012-04-13 00:06:53.969003] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:06:53.969040] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:06:53.970513] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:53.970535] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-client-0 to master-client-3
>[2012-04-13 00:06:53.973957] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file3: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:55.378017] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 15, size: 1500000000, lookups: 36
>[2012-04-13 00:06:55.550447] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:55.550505] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file3: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:55.552937] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:55.552975] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file3 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:55.556632] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file4: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:56.813821] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 37
>[2012-04-13 00:06:57.046289] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:57.046347] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file4: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:57.048852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:57.048890] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file4 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:57.053151] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file9: attempting to move from master-client-1 to master-client-2
>[2012-04-13 00:06:58.557539] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 38
>[2012-04-13 00:06:58.707058] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:06:58.707094] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file9: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:06:58.708580] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:06:58.708602] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file9 from subvolume master-client-1 to master-client-2
>[2012-04-13 00:06:58.719945] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4
>[2012-04-13 00:06:58.722375] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4
>[2012-04-13 00:06:58.727021] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file3: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:00.326570] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 49
>[2012-04-13 00:07:00.420848] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:00.420884] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file3: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:00.422312] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:00.422334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file3 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:00.424782] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file4: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:01.609883] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 50
>[2012-04-13 00:07:01.899096] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:01.899149] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file4: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:01.901393] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:01.901427] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file4 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:01.904949] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file9: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:03.306337] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 51
>[2012-04-13 00:07:03.444197] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:03.444233] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file9: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:03.445642] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:03.445663] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file9 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:03.448698] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:04.784264] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 52
>[2012-04-13 00:07:04.945294] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:04.945329] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:04.946781] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:04.946803] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:04.949265] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:05.849131] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 53
>[2012-04-13 00:07:06.259421] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:06.259457] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:06.260890] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:06.260911] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:06.263339] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:07.348730] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 54
>[2012-04-13 00:07:07.664915] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:07.664972] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:07.667350] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:07.667387] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:07.695387] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5
>[2012-04-13 00:07:07.699706] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5
>[2012-04-13 00:07:07.710098] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file3: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:09.373139] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 65
>[2012-04-13 00:07:09.507392] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:09.507448] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file3: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:09.511316] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:09.511353] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file3 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:09.515571] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file4: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:10.943960] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 25, size: 2500000000, lookups: 66
>[2012-04-13 00:07:11.104956] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:11.105007] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file4: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:11.107175] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:11.107207] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file4 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:11.111133] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file9: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:12.579725] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 26, size: 2600000000, lookups: 67
>[2012-04-13 00:07:13.129100] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:13.129157] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file9: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:13.131836] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:13.131874] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file9 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:13.136773] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:14.300458] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 27, size: 2700000000, lookups: 68
>[2012-04-13 00:07:14.643903] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:14.643953] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:14.646150] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:14.646184] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:14.649563] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:15.948362] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 69
>[2012-04-13 00:07:16.113556] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:16.113599] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:16.115038] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:16.115060] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:16.117261] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:17.212795] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 29, size: 2900000000, lookups: 70
>[2012-04-13 00:07:17.342779] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:17.342813] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:17.344200] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:17.344222] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:17.355682] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6
>[2012-04-13 00:07:17.358103] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6
>[2012-04-13 00:07:17.365385] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:19.016066] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:19.016122] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:19.018497] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:19.018534] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:19.022425] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:20.369374] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 31, size: 3100000000, lookups: 82
>[2012-04-13 00:07:20.754000] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:20.754057] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:20.756480] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:20.756516] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:20.759974] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:21.949229] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 32, size: 3200000000, lookups: 83
>[2012-04-13 00:07:22.070663] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:22.070703] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:22.072215] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:22.072238] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:22.076231] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:23.120173] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:23.120248] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:23.124028] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:23.124066] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:23.127627] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:23.908359] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 34, size: 3400000000, lookups: 85
>[2012-04-13 00:07:24.501183] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:24.501218] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:24.502543] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:24.502564] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:24.505653] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:25.585177] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 35, size: 3500000000, lookups: 86
>[2012-04-13 00:07:25.702059] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:25.702092] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:25.703471] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:25.703493] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:25.714524] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7
>[2012-04-13 00:07:25.716901] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7
>[2012-04-13 00:07:25.721195] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:26.956374] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:26.956419] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:26.958122] W [dht-common.c:945:dht_lookup_everywhere_cbk] 0-master-dht: multiple subvolumes (master-client-3 and master-client-2) have file /large7/file0 (preferably rename the file in the backend, and do a fresh lookup)
>[2012-04-13 00:07:26.958196] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:26.958224] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:26.959883] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file1 on master-client-3
>[2012-04-13 00:07:26.963206] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:28.242774] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:28.242832] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:28.245162] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:28.245200] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:28.247918] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file2 on master-client-3
>[2012-04-13 00:07:28.252473] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:29.536649] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:29.536684] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:29.538088] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:29.538109] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:29.540189] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file3 on master-client-2
>[2012-04-13 00:07:29.542884] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file3: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:30.817169] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:30.817226] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file3: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:30.819590] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:30.819626] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file3 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:30.822001] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file4 on master-client-2
>[2012-04-13 00:07:30.826454] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file4: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:32.329568] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:32.329602] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file4: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:32.330993] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:32.331014] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file4 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:32.332367] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file9 on master-client-2
>[2012-04-13 00:07:32.334982] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file9: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:33.577965] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:33.578020] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file9: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:33.580391] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:33.580428] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file9 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:33.602385] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8
>[2012-04-13 00:07:33.606653] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8
>[2012-04-13 00:07:33.614205] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file3: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:35.177794] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:35.177847] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file3: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:35.180042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file3: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:35.180084] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file3 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:35.183414] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file4: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:36.742035] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:36.742090] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file4: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:36.744423] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file4: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:36.744460] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file4 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:36.748106] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file9: attempting to move from master-client-0 to master-client-2
>[2012-04-13 00:07:38.222033] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory
>[2012-04-13 00:07:38.222090] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file9: failed to perform truncate on master-client-0 (No such file or directory)
>[2012-04-13 00:07:38.224419] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file9: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:38.224455] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file9 from subvolume master-client-0 to master-client-2
>[2012-04-13 00:07:38.229198] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:39.697882] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:39.697916] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:39.699290] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:39.699311] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:39.701450] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:41.158656] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:41.158712] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:41.161105] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:41.161142] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:41.164625] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-client-1 to master-client-3
>[2012-04-13 00:07:42.554399] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory
>[2012-04-13 00:07:42.554457] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-client-1 (No such file or directory)
>[2012-04-13 00:07:42.556840] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory)
>[2012-04-13 00:07:42.556888] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-client-1 to master-client-3
>[2012-04-13 00:07:42.580249] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 48, size: 4800000000, lookups: 128
>[2012-04-13 00:07:42.580686] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 812314: 577299 | 577300