Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 577300 Details for
Bug 812314
Sticky bit is left when you do remove-brick while geo-replication session is active b/w the master and slave
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
maser-rebalance log file .
master-rebalance.log (text/x-log), 432.62 KB, created by
Vijaykumar Koppad
on 2012-04-13 11:05:12 UTC
(
hide
)
Description:
maser-rebalance log file .
Filename:
MIME Type:
Creator:
Vijaykumar Koppad
Created:
2012-04-13 11:05:12 UTC
Size:
432.62 KB
patch
obsolete
>[2012-04-11 21:39:05.569837] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-11 21:39:10.571881] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-11 21:39:10.573190] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-11 21:39:10.577997] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-11 21:39:10.578029] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1' >[2012-04-11 21:39:10.578050] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-11 21:39:10.578068] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-11 21:39:10.578086] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-11 21:39:10.578293] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-11 21:39:10.578405] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-11 21:39:10.589200] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-11 21:39:10.593997] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-11 21:39:10.598650] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-11 21:39:10.603208] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-11 21:39:10.607810] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-11 21:39:10.612435] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 73: end-volume > 74: > 75: volume master-write-behind > 76: type performance/write-behind > 77: subvolumes master-dht > 78: end-volume > 79: > 80: volume master-read-ahead > 81: type performance/read-ahead > 82: subvolumes master-write-behind > 83: end-volume > 84: > 85: volume master-io-cache > 86: type performance/io-cache > 87: subvolumes master-read-ahead > 88: end-volume > 89: > 90: volume master-quick-read > 91: type performance/quick-read > 92: subvolumes master-io-cache > 93: end-volume > 94: > 95: volume master-md-cache > 96: type performance/md-cache > 97: subvolumes master-quick-read > 98: end-volume > 99: >100: volume master >101: type debug/io-stats >102: option latency-measurement off >103: option count-fop-hits off >104: subvolumes master-md-cache >105: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-11 21:39:10.619113] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-11 21:39:10.619275] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-11 21:39:10.619422] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-11 21:39:10.619583] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-11 21:39:10.619726] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24013 (from 0) >[2012-04-11 21:39:10.619904] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24014 (from 0) >[2012-04-11 21:39:10.620059] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-11 21:39:10.620116] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-11 21:39:10.620158] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-11 21:39:10.620197] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-11 21:39:10.620236] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-11 21:39:10.620274] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-11 21:39:14.590132] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-11 21:39:14.590347] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.590790] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-11 21:39:14.590825] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.591050] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-04-11 21:39:14.591244] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-11 21:39:14.595991] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-11 21:39:14.596175] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.596605] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-11 21:39:14.596638] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.597638] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-11 21:39:14.602487] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-11 21:39:14.602655] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.603047] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-11 21:39:14.603083] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.603165] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-11 21:39:14.603368] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-11 21:39:14.608123] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-11 21:39:14.608303] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.608658] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-11 21:39:14.608691] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.609484] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-11 21:39:14.614279] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-11 21:39:14.614445] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.614840] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24013, attached to remote volume '/exportdir/master/d5'. >[2012-04-11 21:39:14.614875] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.614956] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-11 21:39:14.615140] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-11 21:39:14.619909] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-11 21:39:14.620073] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:39:14.620453] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24014, attached to remote volume '/exportdir/master/d6'. >[2012-04-11 21:39:14.620486] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:39:14.621319] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-11 21:39:14.641034] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-11 21:39:14.644048] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-11 21:39:14.644203] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-11 21:39:14.644460] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-11 21:39:14.652888] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-11 21:39:14.671114] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-11 21:39:14.680396] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-11 21:39:14.693968] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:15.938269] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 1 >[2012-04-11 21:39:17.994659] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:17.994699] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:17.994769] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:17.994798] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:17.995336] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:18.000429] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:18.000466] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:18.009214] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:19.553575] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 100000000, lookups: 2 >[2012-04-11 21:39:21.749250] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 1, size: 100000000, lookups: 2 >[2012-04-11 21:39:22.366777] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:22.366814] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:22.366854] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:22.366876] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:22.370247] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:22.378208] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:22.378244] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:22.389262] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:23.710172] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 200000000, lookups: 3 >[2012-04-11 21:39:25.986970] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 200000000, lookups: 3 >[2012-04-11 21:39:26.157536] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:26.157575] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:26.157612] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:26.157634] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:26.162106] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:26.173790] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:26.173832] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:26.261196] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-11 21:39:26.290715] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-11 21:39:26.324693] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:27.979940] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 14 >[2012-04-11 21:39:29.654944] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 14 >[2012-04-11 21:39:29.843913] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:29.843950] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:29.843988] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:29.844011] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:29.844453] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:29.849726] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:29.849793] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:29.854900] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:31.577233] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 400000000, lookups: 15 >[2012-04-11 21:39:33.286935] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 400000000, lookups: 15 >[2012-04-11 21:39:34.254543] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:34.254580] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:34.254618] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:34.254641] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:34.255108] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:34.263628] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:34.263667] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:34.270923] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:37.014401] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 5, size: 500000000, lookups: 16 >[2012-04-11 21:39:38.002869] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:38.002908] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:38.002947] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:38.002970] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:38.003398] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:38.008594] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:38.008632] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:38.037804] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-11 21:39:38.048777] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-11 21:39:38.066058] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:38.863035] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 27 >[2012-04-11 21:39:41.341718] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 27 >[2012-04-11 21:39:41.773887] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:41.773927] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:41.773969] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:41.773992] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:41.774446] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:41.779366] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:41.779403] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:41.784379] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:43.204921] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 28 >[2012-04-11 21:39:44.622875] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 28 >[2012-04-11 21:39:45.041710] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:45.041757] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:45.041785] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:45.041799] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:45.042064] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:45.045656] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:45.045681] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:45.049330] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:48.021014] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:48.021041] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:39:48.021064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:48.021077] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:39:48.021347] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:39:48.024378] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:48.024400] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:39:48.042324] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-11 21:39:48.048213] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-11 21:39:48.062101] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:50.811403] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:50.811430] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:39:50.811454] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:50.811467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:39:50.811731] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:39:50.814720] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:50.814770] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:50.818091] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:53.456307] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:53.456333] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:39:53.456355] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:53.456377] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:39:53.456626] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:39:53.459492] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:53.459513] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:53.462777] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:56.194607] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:56.194633] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:39:56.194656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:56.194669] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:39:56.194942] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:39:56.197853] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:56.197875] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:56.217411] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-11 21:39:56.227532] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-11 21:39:56.245717] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:58.914576] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:58.914603] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:39:58.914626] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:39:58.914640] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:39:58.914923] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:39:58.918054] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:39:58.918076] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:39:58.921340] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:01.899675] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:01.899702] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:01.899725] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:01.899758] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:01.900018] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:01.902980] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:01.903002] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:01.906349] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:04.816933] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:04.816960] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:04.816983] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:04.816996] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:04.817250] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:04.820230] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:04.820251] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:04.835495] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-11 21:40:04.840857] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-11 21:40:04.854492] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:07.711953] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:07.711981] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:07.712005] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:07.712018] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:07.712295] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:07.715255] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:07.715277] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:07.719190] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:10.868821] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:10.868859] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:10.868898] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:10.868920] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:10.869344] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:10.874287] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:10.874323] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:10.879292] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:14.458031] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:14.458070] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:14.458108] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:14.458130] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:14.458580] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:14.463548] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:14.463585] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:14.485259] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-11 21:40:14.494716] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-11 21:40:14.509638] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:17.800729] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:17.800799] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:40:17.800841] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:17.800873] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:40:17.801302] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:40:17.806319] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:17.806356] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:17.811426] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:21.118549] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:21.118588] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:40:21.118626] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:21.118648] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:40:21.119125] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:40:21.128908] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:21.128951] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:21.139075] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:24.349520] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:24.349572] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 21:40:24.349627] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:24.349653] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 21:40:24.350216] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 21:40:24.358303] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:24.358338] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 21:40:24.389809] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-11 21:40:24.399070] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-11 21:40:24.422680] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:27.808676] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:27.808715] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:27.808798] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:27.808826] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:27.809257] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:27.814241] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:27.814277] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:27.820215] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:31.002968] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:31.003005] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:31.003045] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:31.003068] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:31.003502] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:31.008589] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:31.008627] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:31.014210] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:34.320123] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:34.320160] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 21:40:34.320197] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:40:34.320218] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 21:40:34.320629] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 21:40:34.328337] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:40:34.328371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 21:40:34.359478] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 104 >[2012-04-11 21:40:34.360009] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-11 21:45:07.471622] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-11 21:45:12.473586] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-11 21:45:12.473661] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-11 21:45:12.560459] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-11 21:45:12.560485] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5' >[2012-04-11 21:45:12.560497] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-11 21:45:12.560507] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-11 21:45:12.560517] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-11 21:45:12.560643] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-11 21:45:12.560711] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-11 21:45:12.560825] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2 >[2012-04-11 21:45:13.004366] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-11 21:45:13.007066] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-11 21:45:13.009527] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-11 21:45:13.011948] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-11 21:45:13.014344] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-11 21:45:13.016827] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: option decommissioned-bricks master-replicate-2 > 73: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 74: end-volume > 75: > 76: volume master-write-behind > 77: type performance/write-behind > 78: subvolumes master-dht > 79: end-volume > 80: > 81: volume master-read-ahead > 82: type performance/read-ahead > 83: subvolumes master-write-behind > 84: end-volume > 85: > 86: volume master-io-cache > 87: type performance/io-cache > 88: subvolumes master-read-ahead > 89: end-volume > 90: > 91: volume master-quick-read > 92: type performance/quick-read > 93: subvolumes master-io-cache > 94: end-volume > 95: > 96: volume master-md-cache > 97: type performance/md-cache > 98: subvolumes master-quick-read > 99: end-volume >100: >101: volume master >102: type debug/io-stats >103: option latency-measurement off >104: option count-fop-hits off >105: subvolumes master-md-cache >106: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-11 21:45:13.020553] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-11 21:45:13.020663] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-11 21:45:13.020771] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-11 21:45:13.020860] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-11 21:45:13.020946] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24013 (from 0) >[2012-04-11 21:45:13.021031] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24014 (from 0) >[2012-04-11 21:45:13.021124] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-11 21:45:13.021159] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-11 21:45:13.021183] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-11 21:45:13.021205] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-11 21:45:13.021227] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-11 21:45:13.021248] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-11 21:45:16.487882] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-11 21:45:16.488084] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.493173] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-11 21:45:16.493335] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.498904] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-11 21:45:16.499068] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.502339] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-11 21:45:16.502375] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.611686] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-11 21:45:16.611849] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-11 21:45:16.611881] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.612185] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-04-11 21:45:16.612285] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-11 21:45:16.612316] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.613321] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-11 21:45:16.613377] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-11 21:45:16.613429] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-11 21:45:16.613625] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-11 21:45:16.613728] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.614188] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-11 21:45:16.614221] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.615154] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-11 21:45:16.617941] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-11 21:45:16.618004] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.618367] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24013, attached to remote volume '/exportdir/master/d5'. >[2012-04-11 21:45:16.618399] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.618656] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-11 21:45:16.618716] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-11 21:45:16.621189] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-11 21:45:16.621367] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 21:45:16.621938] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24014, attached to remote volume '/exportdir/master/d6'. >[2012-04-11 21:45:16.621984] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 21:45:16.623267] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-11 21:45:16.706355] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-11 21:45:16.709613] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-11 21:45:16.709794] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-11 21:45:16.710044] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-11 21:45:16.718646] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-11 21:45:16.737361] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-11 21:45:16.746951] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-11 21:45:16.794287] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:20.909698] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:20.909767] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:20.909816] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:20.909839] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:20.910267] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:20.917556] W [dht-common.c:945:dht_lookup_everywhere_cbk] 0-master-dht: multiple subvolumes (master-replicate-0 and master-replicate-1) have file /large1/file0 (preferably rename the file in the backend, and do a fresh lookup) >[2012-04-11 21:45:20.918689] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:20.918726] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:20.927645] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large1/file1 on master-replicate-1 >[2012-04-11 21:45:20.940863] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:24.899449] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:24.899475] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:24.899498] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:24.899511] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:24.899781] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:24.902833] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:24.902855] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:24.905000] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large1/file2 on master-replicate-1 >[2012-04-11 21:45:24.913856] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:28.549245] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:28.549275] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:28.549298] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:28.549322] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:28.549582] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:28.552528] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:28.552549] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:28.561353] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-11 21:45:28.566565] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-11 21:45:28.584566] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:28.820456] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 300000000, lookups: 18 >[2012-04-11 21:45:31.343510] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:31.343537] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:31.343560] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:31.343573] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:31.343846] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:31.346687] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:31.346708] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:31.349692] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:34.124531] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:34.124558] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:34.124580] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:34.124593] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:34.126024] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:34.133147] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:34.133179] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:34.136384] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:37.089609] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:37.089636] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:37.089659] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:37.089672] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:37.089942] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:37.092907] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:37.092930] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:37.099718] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-11 21:45:37.104967] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-11 21:45:37.120711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:40.752657] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:40.752685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:40.752709] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:40.752722] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:40.753240] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:40.759474] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:40.759496] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:40.769529] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:43.418196] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:43.418224] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:43.418252] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:43.418266] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:43.418539] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:43.421523] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:43.421545] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:43.427171] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:46.229057] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:46.229084] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:46.229106] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:46.229119] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:46.229381] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:46.232324] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:46.232346] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:45:46.239145] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-11 21:45:46.244511] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-11 21:45:46.260170] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:48.967314] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:48.967341] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:48.967364] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:48.967397] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:48.967682] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:48.970728] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:48.970767] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:48.982688] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:51.980446] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:51.980472] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:51.980495] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:51.980507] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:51.980791] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:51.983723] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:51.983762] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:51.989372] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:54.589678] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:54.589704] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:54.589727] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:54.589761] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:54.590018] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:54.598571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:54.598594] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:54.610921] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-11 21:45:54.620383] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-11 21:45:54.655676] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:57.384754] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:57.384781] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:45:57.384805] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:45:57.384818] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:45:57.385071] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:45:57.388018] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:45:57.388040] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:45:57.393582] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:00.091807] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:00.091834] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:00.091855] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:00.091874] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:00.092130] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:00.095313] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:00.095334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:00.101119] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:02.910504] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:02.910532] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:02.910555] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:02.910568] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:02.910855] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:02.913836] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:02.913857] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:02.920650] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-11 21:46:02.925950] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-11 21:46:02.941791] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:05.660484] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:05.660510] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:05.660534] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:05.660547] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:05.660831] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:05.663795] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:05.663816] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:05.669644] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:08.439593] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:08.439619] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:08.439654] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:08.439667] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:08.439938] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:08.442888] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:08.442914] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:08.448679] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:11.144548] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:11.144574] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:11.144607] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:11.144621] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:11.144888] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:11.147807] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:11.147833] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:11.154436] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-11 21:46:11.159682] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-11 21:46:11.175727] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:14.133404] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:14.133431] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:14.133453] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:14.133466] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:14.133724] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:14.136575] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:14.136606] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:14.142057] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:14.662099] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 69 >[2012-04-11 21:46:16.599375] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 69 >[2012-04-11 21:46:16.998442] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:16.998467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:16.998490] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:16.998503] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:16.998771] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:17.001638] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:17.001659] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:17.007097] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:18.124543] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 70 >[2012-04-11 21:46:19.450415] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 70 >[2012-04-11 21:46:19.758251] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:19.758278] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:19.758301] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:19.758314] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:19.758573] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:19.761511] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:19.761532] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-11 21:46:19.768246] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-11 21:46:19.773361] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-11 21:46:19.788479] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:22.438477] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:22.438503] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:22.438525] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:22.438538] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:22.438804] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:22.441672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:22.441693] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:22.447401] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:25.183283] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:25.183309] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:25.183332] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:25.183345] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:25.183601] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:25.186499] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:25.186520] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:25.192017] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:27.943389] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:27.943416] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-11 21:46:27.943438] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 21:46:27.943451] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-11 21:46:27.943714] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-11 21:46:27.946638] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 21:46:27.946660] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-11 21:46:27.953202] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 80 >[2012-04-11 21:46:27.953518] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-11 22:04:07.447208] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-11 22:04:12.448572] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-11 22:04:12.448631] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-11 22:04:12.450774] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-11 22:04:12.450804] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1' >[2012-04-11 22:04:12.450817] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-11 22:04:12.450828] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-11 22:04:12.450838] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-11 22:04:12.450983] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-11 22:04:12.451052] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-11 22:04:12.460273] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-11 22:04:12.462966] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-11 22:04:12.465318] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-11 22:04:12.467676] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-11 22:04:12.470120] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-11 22:04:12.472481] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 73: end-volume > 74: > 75: volume master-write-behind > 76: type performance/write-behind > 77: subvolumes master-dht > 78: end-volume > 79: > 80: volume master-read-ahead > 81: type performance/read-ahead > 82: subvolumes master-write-behind > 83: end-volume > 84: > 85: volume master-io-cache > 86: type performance/io-cache > 87: subvolumes master-read-ahead > 88: end-volume > 89: > 90: volume master-quick-read > 91: type performance/quick-read > 92: subvolumes master-io-cache > 93: end-volume > 94: > 95: volume master-md-cache > 96: type performance/md-cache > 97: subvolumes master-quick-read > 98: end-volume > 99: >100: volume master >101: type debug/io-stats >102: option latency-measurement off >103: option count-fop-hits off >104: subvolumes master-md-cache >105: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-11 22:04:12.476177] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-11 22:04:12.476280] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-11 22:04:12.476377] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-11 22:04:12.476464] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-11 22:04:12.476549] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24015 (from 0) >[2012-04-11 22:04:12.476635] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24016 (from 0) >[2012-04-11 22:04:12.476728] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-11 22:04:12.476784] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-11 22:04:12.476809] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-11 22:04:12.476831] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-11 22:04:12.476854] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-11 22:04:12.476875] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-11 22:04:16.466525] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-11 22:04:16.466725] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.467157] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-11 22:04:16.467188] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.467512] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-04-11 22:04:16.467696] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-11 22:04:16.472312] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-11 22:04:16.472466] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.472862] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-11 22:04:16.472895] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.473777] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-11 22:04:16.478607] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-11 22:04:16.478797] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.479150] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-11 22:04:16.479183] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.479263] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-11 22:04:16.479439] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-11 22:04:16.484162] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-11 22:04:16.484325] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.484701] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-11 22:04:16.484766] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.485536] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-11 22:04:16.490390] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-11 22:04:16.490584] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.490996] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24015, attached to remote volume '/exportdir/master/d5'. >[2012-04-11 22:04:16.491030] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.491113] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-11 22:04:16.491297] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-11 22:04:16.496030] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-11 22:04:16.496194] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-11 22:04:16.496576] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24016, attached to remote volume '/exportdir/master/d6'. >[2012-04-11 22:04:16.496609] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-11 22:04:16.497440] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-11 22:04:16.514402] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-11 22:04:16.517774] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-11 22:04:16.517923] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-11 22:04:16.518165] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-11 22:04:16.526840] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-11 22:04:16.545912] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-11 22:04:16.560565] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-11 22:04:16.573629] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:20.292580] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:20.292606] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:20.292629] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:20.292642] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:20.292920] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:20.295857] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:20.295878] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:20.297905] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:24.797634] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:24.797672] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:24.797711] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:24.797763] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:24.798197] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:24.807888] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:24.807923] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:24.831998] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:28.379440] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:28.379467] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:28.379490] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:28.379507] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:28.379786] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:28.382967] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:28.382990] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:28.385943] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:32.177437] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:32.177476] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:32.177514] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:32.177537] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:32.178026] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:32.183026] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:32.183063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:32.188009] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:36.083605] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:36.083632] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:36.083655] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:36.083669] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:36.083956] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:36.086913] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:36.086935] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:36.090330] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:39.037660] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:39.037688] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:39.037712] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:39.037726] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:39.038006] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:39.041008] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:39.041030] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:04:39.046153] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file3: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:41.732430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:41.732458] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:04:41.732481] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:41.732493] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:04:41.732759] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file3: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:04:41.735672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:41.735697] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file3 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:41.738952] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file4: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:44.409656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:44.409684] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:04:44.409708] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:44.409722] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:04:44.409997] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file4: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:04:44.412982] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:44.413005] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file4 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:44.416469] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file7: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:47.022877] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:47.022904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:04:47.022926] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:47.022940] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:04:47.023189] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file7: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:04:47.026112] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:47.026133] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file7 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:47.029387] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file9: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:49.742147] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:49.742174] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:04:49.742196] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:49.742209] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:04:49.742454] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file9: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:04:49.745328] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:49.745349] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file9 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:04:49.757494] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-11 22:04:49.762629] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-11 22:04:49.773372] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:52.438035] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:52.438061] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:52.438083] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:52.438096] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:52.438352] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:52.441936] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:52.441957] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:52.445308] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:55.182358] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:55.182383] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:55.182415] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:55.182429] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:55.182675] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:55.185611] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:55.185632] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:55.188917] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:57.887546] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:57.887588] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:04:57.887616] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:04:57.887634] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:04:57.887921] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:04:57.891530] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:04:57.891552] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:04:57.905707] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-11 22:04:57.910962] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-11 22:04:57.916853] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large3/file5 on master-replicate-1 >[2012-04-11 22:04:57.922803] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:00.644687] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:00.644727] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:00.644802] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:00.644827] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:00.645252] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:00.650462] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:00.650512] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:00.655997] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:03.398381] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:03.398408] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:03.398430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:03.398443] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:03.398711] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:03.402610] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:03.402634] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:03.406514] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:06.036842] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:06.036868] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:06.036891] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:06.036904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:06.037156] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:06.040099] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:06.040120] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:06.043468] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:08.820160] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:08.820186] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:08.820209] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:08.820223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:08.820489] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:08.823592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:08.823618] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:08.826993] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:11.531675] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:11.531701] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:11.531724] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:11.531759] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:11.532027] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:11.540761] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:11.540795] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:11.544342] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:14.254845] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:14.254872] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:14.254897] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:14.254911] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:14.255173] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:14.258245] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:14.258269] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:05:14.270495] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file3: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:17.427195] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:17.427223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:17.427247] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:17.427261] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:17.427534] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file3: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:17.430679] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:17.430704] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file3 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:17.434592] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file4: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:20.136482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:20.136510] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:20.136534] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:20.136547] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:20.136819] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file4: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:20.139787] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:20.139823] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file4 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:20.142907] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file7: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:22.826287] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:22.826313] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:22.826335] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:22.826348] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:22.826654] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file7: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:22.829577] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:22.829598] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file7 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:22.832859] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file9: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:25.555481] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:25.555506] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:25.555529] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:25.555542] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:25.555807] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file9: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:25.558779] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:25.558800] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file9 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:25.571043] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-11 22:05:25.576225] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-11 22:05:25.588947] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:28.280132] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:28.280158] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:28.280181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:28.280194] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:28.280458] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:28.283465] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:28.283487] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:28.286900] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:31.228011] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:31.228037] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:31.228060] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:31.228073] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:31.228321] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:31.235906] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:31.235930] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:31.239463] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:33.868689] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:33.868715] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:33.868757] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:33.868774] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:33.869022] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:33.876591] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:33.876612] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:33.899010] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-11 22:05:33.921160] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-11 22:05:33.945003] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:36.423172] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:36.423198] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:36.423220] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:36.423238] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:36.423526] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:36.427157] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:36.427179] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:36.430803] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:38.998375] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:38.998408] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:38.998433] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:38.998447] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:38.998693] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:39.001592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:39.001612] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:39.004932] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:41.895907] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:41.895933] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:41.895956] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:41.895968] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:41.896226] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:41.900557] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:41.900579] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:05:41.912595] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-11 22:05:41.922098] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-11 22:05:41.934337] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:44.633955] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:44.633982] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:44.634007] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:44.634029] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:44.637512] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:44.641571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:44.641601] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:44.645977] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:47.243064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:47.243090] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:47.243113] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:47.243125] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:47.246635] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:47.250499] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:47.250519] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:47.255013] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:49.849070] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:49.849096] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:49.849118] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:49.849131] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:49.849897] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:49.857510] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:49.857533] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:49.864434] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:52.434107] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:52.434136] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:05:52.434160] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:52.434183] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:05:52.434459] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:05:52.442429] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:52.442456] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:05:52.451666] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:55.247204] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:55.247243] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:55.247289] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:55.247307] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:55.247575] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:55.251386] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:55.251408] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:55.254976] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:58.315972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:58.316000] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:05:58.316024] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:05:58.316037] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:05:58.316318] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:05:58.319423] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:05:58.319448] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:05:58.323077] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:01.156870] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:01.156895] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:01.156918] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:01.156931] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:01.157197] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:01.160180] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:01.160202] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:01.171157] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-11 22:06:01.176407] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-11 22:06:01.187326] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:03.884952] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:03.884978] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:03.885000] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:03.885012] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:03.885261] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:03.888346] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:03.888368] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:03.891784] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:06.465185] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:06.465210] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:06.465232] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:06.465245] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:06.465497] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:06.468476] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:06.468498] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:06.471821] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:09.133660] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:09.133685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:09.133708] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:09.133728] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:09.133998] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:09.137134] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:09.137155] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-11 22:06:09.150832] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-11 22:06:09.155805] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-11 22:06:09.162711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file3: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:11.952428] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:11.952455] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:11.952477] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:11.952491] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:11.952760] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file3: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:11.955797] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:11.955819] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file3 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:11.959383] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file4: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:14.617348] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:14.617375] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:14.617398] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:14.617411] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:14.617671] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file4: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:14.620820] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:14.620842] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file4 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:14.624201] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file7: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:17.357807] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:17.357835] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:17.357868] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:17.357882] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:17.358137] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file7: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:17.361118] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:17.361140] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file7 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:17.368270] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file9: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:20.222037] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:20.222067] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-11 22:06:20.222090] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:20.222103] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-11 22:06:20.222370] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file9: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-11 22:06:20.225176] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:20.225198] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file9 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-11 22:06:20.230802] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file5: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:22.999380] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:22.999406] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:22.999428] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:22.999441] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:22.999693] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:23.002794] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:23.002816] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file5 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:23.006186] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file6: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:25.824391] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:25.824417] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:25.824451] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:25.824475] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:25.824730] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:25.827721] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:25.827760] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file6 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:25.831113] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file8: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:28.473273] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:28.473298] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:28.473321] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:28.473334] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:28.473596] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:28.476603] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:28.476625] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file8 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-11 22:06:28.480210] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:31.096126] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:31.096152] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:31.096175] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:31.096188] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:31.096441] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:31.100080] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:31.100102] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:31.103635] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:33.815078] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:33.815105] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:33.815128] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:33.815148] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:33.815421] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:33.821044] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:33.821067] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:33.824588] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:36.406969] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:36.406996] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-11 22:06:36.407019] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-11 22:06:36.407033] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-11 22:06:36.407302] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-11 22:06:36.410762] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-11 22:06:36.410787] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-11 22:06:36.425111] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 49, size: 4900000000, lookups: 115 >[2012-04-11 22:06:36.425465] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-12 20:26:00.009545] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-12 20:26:05.033233] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-12 20:26:05.036009] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-12 20:26:05.111797] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-12 20:26:05.111835] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5' >[2012-04-12 20:26:05.111856] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-12 20:26:05.111875] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-12 20:26:05.112247] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-12 20:26:05.112486] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-12 20:26:05.112598] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-12 20:26:05.112789] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2 >[2012-04-12 20:26:05.143396] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-12 20:26:05.148170] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-12 20:26:05.154083] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-12 20:26:05.158680] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-12 20:26:05.163422] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-12 20:26:05.168174] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: option decommissioned-bricks master-replicate-2 > 73: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 74: end-volume > 75: > 76: volume master-write-behind > 77: type performance/write-behind > 78: subvolumes master-dht > 79: end-volume > 80: > 81: volume master-read-ahead > 82: type performance/read-ahead > 83: subvolumes master-write-behind > 84: end-volume > 85: > 86: volume master-io-cache > 87: type performance/io-cache > 88: subvolumes master-read-ahead > 89: end-volume > 90: > 91: volume master-quick-read > 92: type performance/quick-read > 93: subvolumes master-io-cache > 94: end-volume > 95: > 96: volume master-md-cache > 97: type performance/md-cache > 98: subvolumes master-quick-read > 99: end-volume >100: >101: volume master >102: type debug/io-stats >103: option latency-measurement off >104: option count-fop-hits off >105: subvolumes master-md-cache >106: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-12 20:26:05.174890] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-12 20:26:05.175057] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-12 20:26:05.175202] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-12 20:26:05.175362] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-12 20:26:05.175504] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24015 (from 0) >[2012-04-12 20:26:05.175646] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24016 (from 0) >[2012-04-12 20:26:05.175870] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-12 20:26:05.175929] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-12 20:26:05.175971] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-12 20:26:05.176009] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-12 20:26:05.176047] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-12 20:26:05.176085] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-12 20:26:09.066177] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-12 20:26:09.070917] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-12 20:26:09.075709] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-12 20:26:09.080431] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-12 20:26:09.085145] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-12 20:26:09.087896] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.088065] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.088173] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.088277] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.088404] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.091892] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-12 20:26:09.092069] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:26:09.110317] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-12 20:26:09.110352] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.110659] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-12 20:26:09.110727] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-12 20:26:09.110796] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.110887] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-04-12 20:26:09.110946] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-12 20:26:09.110969] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.112408] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-12 20:26:09.112442] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.113271] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-12 20:26:09.113332] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24015, attached to remote volume '/exportdir/master/d5'. >[2012-04-12 20:26:09.113356] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.113607] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-12 20:26:09.113664] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-12 20:26:09.113702] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-12 20:26:09.113777] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24016, attached to remote volume '/exportdir/master/d6'. >[2012-04-12 20:26:09.113805] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:26:09.114662] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-12 20:26:09.114724] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-12 20:26:09.114823] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-12 20:26:09.125140] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-12 20:26:09.128189] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-12 20:26:09.128329] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-12 20:26:09.145197] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-12 20:26:09.167863] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-12 20:26:09.195873] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-12 20:26:09.205499] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-12 20:26:09.260040] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:14.732047] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:14.747961] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:14.748018] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:14.748053] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:14.748539] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:14.753694] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:14.753730] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:14.763711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:18.904931] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:18.904972] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:18.905011] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:18.905034] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:18.905478] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:18.911571] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:18.911607] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:18.916598] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:21.875703] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:21.875775] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:21.875822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:21.875846] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:21.876272] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:21.881852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:21.881889] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:21.893505] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-12 20:26:21.902717] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-12 20:26:22.147508] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:25.817787] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:25.817829] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:25.817886] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:25.817910] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:25.818335] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:25.827265] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:25.827303] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:25.835722] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:28.972723] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:28.972795] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:28.976710] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:28.976779] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:28.977218] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:28.982042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:28.982079] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:28.987120] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:32.054278] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:32.054320] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:32.054361] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:32.054384] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:32.054859] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:32.059815] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:32.059852] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:26:32.071489] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-12 20:26:32.096449] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-12 20:26:32.243951] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:35.640953] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:35.641005] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:35.641047] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:35.641069] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:35.641455] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:35.647809] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:35.647839] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:35.652280] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:38.708734] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:38.708804] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:38.708847] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:38.708871] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:38.709315] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:38.714252] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:38.714289] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:38.719349] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:41.906924] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:41.906963] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:41.907002] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:41.907025] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:41.907472] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:41.912422] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:41.912459] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:41.924123] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-12 20:26:41.942845] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-12 20:26:42.149156] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:45.238800] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:45.238841] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:45.238881] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:45.238904] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:45.239333] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:45.249829] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:45.249868] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:45.255164] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:49.019833] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:49.019873] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:49.019913] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:49.019936] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:49.020382] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:49.025339] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:49.025376] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:49.030506] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:52.802267] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:52.802305] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:52.802345] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:52.802368] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:52.802841] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:52.811699] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:52.811767] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:52.834163] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-12 20:26:52.848951] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-12 20:26:52.919144] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:57.544461] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:57.544504] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:26:57.544543] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:26:57.544567] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:26:57.545028] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:26:57.556919] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:26:57.556957] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:26:57.566627] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:27:00.751037] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:00.751079] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:00.751119] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:00.751142] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:00.751587] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:00.756541] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:00.756578] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:27:00.761705] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:27:03.991763] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:03.991791] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:03.991814] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:03.991827] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:03.992079] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:03.994848] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:03.994870] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:27:04.001539] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-12 20:27:04.006724] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-12 20:27:04.046685] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:07.290509] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:07.290549] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:27:07.290589] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:07.290612] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:27:07.291070] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:27:07.296145] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:07.296182] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:07.301321] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:10.747532] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:10.747576] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:27:10.762482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:10.762518] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:27:10.762987] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:27:10.768026] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:10.768063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:10.773206] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:13.787114] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:13.787154] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:27:13.787193] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:13.787217] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:27:13.787659] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:27:13.792673] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:13.792709] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:27:13.809539] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:17.143891] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:17.143917] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:17.143941] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:17.143954] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:17.144207] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:17.147139] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:17.147161] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:17.150328] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:20.444851] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:20.444891] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:20.444931] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:20.444954] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:20.445400] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:20.450449] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:20.450485] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:20.455772] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:23.110333] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:23.110361] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:23.110397] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:23.110411] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:23.110656] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:23.113553] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:23.113575] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:23.116674] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:26.270671] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:26.270709] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:26.270781] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:26.270810] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:26.271240] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:26.276272] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:26.276307] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:26.288085] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-12 20:27:26.297215] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-12 20:27:26.365204] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:29.760181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:29.760221] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:29.760260] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:29.760283] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:29.760722] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:29.765719] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:29.765785] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:29.770926] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:32.965305] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:32.965347] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:32.965656] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:32.965688] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:32.966146] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:32.970984] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:32.971036] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:32.976162] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:36.460009] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:36.460050] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:36.460090] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:36.460114] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:36.460555] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:36.465707] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:36.465773] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:36.477544] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-12 20:27:36.486646] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-12 20:27:36.544917] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:39.693023] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:39.693057] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:39.693095] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:39.693116] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:39.693480] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:39.696516] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:39.696537] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:39.699518] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:42.597229] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:42.597272] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:42.597312] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:42.597335] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:42.597809] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:42.604328] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:42.604364] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:42.609447] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:45.613252] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:45.613291] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:27:45.613332] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:27:45.613356] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:27:45.613825] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:27:45.618761] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:27:45.618799] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:27:45.671517] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 83 >[2012-04-12 20:27:45.812429] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-12 20:30:05.335288] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-12 20:30:10.360597] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-12 20:30:10.360677] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-12 20:30:10.365924] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-12 20:30:10.365958] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1' >[2012-04-12 20:30:10.365980] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-12 20:30:10.365999] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-12 20:30:10.366019] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-12 20:30:10.366222] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-12 20:30:10.366355] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-12 20:30:10.373998] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-12 20:30:10.378887] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-12 20:30:10.383538] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-12 20:30:10.388724] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-replicate-0 > 38: type cluster/replicate > 39: subvolumes master-client-0 master-client-1 > 40: end-volume > 41: > 42: volume master-replicate-1 > 43: type cluster/replicate > 44: subvolumes master-client-2 master-client-3 > 45: end-volume > 46: > 47: volume master-dht > 48: type cluster/distribute > 49: subvolumes master-replicate-0 master-replicate-1 > 50: end-volume > 51: > 52: volume master-write-behind > 53: type performance/write-behind > 54: subvolumes master-dht > 55: end-volume > 56: > 57: volume master-read-ahead > 58: type performance/read-ahead > 59: subvolumes master-write-behind > 60: end-volume > 61: > 62: volume master-io-cache > 63: type performance/io-cache > 64: subvolumes master-read-ahead > 65: end-volume > 66: > 67: volume master-quick-read > 68: type performance/quick-read > 69: subvolumes master-io-cache > 70: end-volume > 71: > 72: volume master-md-cache > 73: type performance/md-cache > 74: subvolumes master-quick-read > 75: end-volume > 76: > 77: volume master > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes master-md-cache > 82: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-12 20:30:10.395605] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-12 20:30:10.395800] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-12 20:30:10.395952] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-12 20:30:10.396095] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-12 20:30:10.396250] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-12 20:30:10.396324] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-12 20:30:10.396369] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-12 20:30:10.396408] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-12 20:30:14.357429] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-12 20:30:14.357631] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:30:14.358081] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-12 20:30:14.358132] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:30:14.358648] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-04-12 20:30:14.358878] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-12 20:30:14.363823] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-12 20:30:14.363990] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:30:14.364347] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-12 20:30:14.364396] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:30:14.365385] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-12 20:30:14.370122] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-12 20:30:14.370283] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:30:14.370650] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-12 20:30:14.370683] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:30:14.370808] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-12 20:30:14.370997] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-12 20:30:14.375687] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-12 20:30:14.375879] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:30:14.376264] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-12 20:30:14.376303] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:30:14.377183] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-12 20:30:14.396696] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-12 20:30:14.399077] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-12 20:30:14.399343] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-12 20:30:14.405131] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-12 20:30:14.418664] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-12 20:30:14.425138] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-12 20:30:14.454548] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-12 20:30:14.461089] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-12 20:30:14.490450] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-12 20:30:14.497065] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-12 20:30:14.526405] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-12 20:30:14.532919] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-12 20:30:14.561986] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-12 20:30:14.568239] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-12 20:30:14.597112] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-12 20:30:14.603391] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-12 20:30:14.631910] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-12 20:30:14.638100] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-12 20:30:14.666473] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-12 20:30:14.672683] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-12 20:30:14.700076] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 80 >[2012-04-12 20:30:14.717664] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-12 20:39:42.127518] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-12 20:39:47.129680] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-12 20:39:47.130877] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-12 20:39:47.164946] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-12 20:39:47.164987] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1' >[2012-04-12 20:39:47.165008] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-12 20:39:47.165028] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-12 20:39:47.165047] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-12 20:39:47.165276] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-12 20:39:47.165389] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-12 20:39:47.229102] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-12 20:39:47.233982] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-12 20:39:47.238605] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-12 20:39:47.243102] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-12 20:39:47.247606] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-12 20:39:47.252138] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 73: end-volume > 74: > 75: volume master-write-behind > 76: type performance/write-behind > 77: subvolumes master-dht > 78: end-volume > 79: > 80: volume master-read-ahead > 81: type performance/read-ahead > 82: subvolumes master-write-behind > 83: end-volume > 84: > 85: volume master-io-cache > 86: type performance/io-cache > 87: subvolumes master-read-ahead > 88: end-volume > 89: > 90: volume master-quick-read > 91: type performance/quick-read > 92: subvolumes master-io-cache > 93: end-volume > 94: > 95: volume master-md-cache > 96: type performance/md-cache > 97: subvolumes master-quick-read > 98: end-volume > 99: >100: volume master >101: type debug/io-stats >102: option latency-measurement off >103: option count-fop-hits off >104: subvolumes master-md-cache >105: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-12 20:39:47.258862] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-12 20:39:47.259033] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-12 20:39:47.259180] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-12 20:39:47.259325] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-12 20:39:47.259485] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24017 (from 0) >[2012-04-12 20:39:47.259631] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24018 (from 0) >[2012-04-12 20:39:47.259819] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-12 20:39:47.259879] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-12 20:39:47.259931] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-12 20:39:47.259973] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-12 20:39:47.260012] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-12 20:39:47.260049] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-12 20:39:51.145395] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-12 20:39:51.145569] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.146481] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-12 20:39:51.146515] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.146833] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-0' came back up; going online. >[2012-04-12 20:39:51.147027] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-12 20:39:51.151374] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-12 20:39:51.151532] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.151915] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-12 20:39:51.151950] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.152789] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-12 20:39:51.157445] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-12 20:39:51.157600] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.157986] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-12 20:39:51.158021] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.158106] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-2' came back up; going online. >[2012-04-12 20:39:51.158306] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-12 20:39:51.162978] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-12 20:39:51.163134] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.163500] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-12 20:39:51.163533] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.164306] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-12 20:39:51.169053] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-12 20:39:51.169209] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.169572] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24017, attached to remote volume '/exportdir/master/d5'. >[2012-04-12 20:39:51.169615] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.169698] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-12 20:39:51.169916] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-12 20:39:51.174607] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-12 20:39:51.174798] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:39:51.175160] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24018, attached to remote volume '/exportdir/master/d6'. >[2012-04-12 20:39:51.175193] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:39:51.176053] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-12 20:39:51.188166] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-12 20:39:51.191427] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-12 20:39:51.191575] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-12 20:39:51.191856] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-12 20:39:51.200429] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-12 20:39:51.219418] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-12 20:39:51.229238] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-12 20:39:51.250526] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:39:54.692654] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:39:54.692691] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:39:54.692728] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:39:54.692783] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:39:54.693188] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:39:54.697887] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:39:54.697920] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:39:54.701005] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:39:58.382487] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:39:58.382526] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:39:58.382567] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:39:58.382600] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:39:58.383071] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:39:58.388051] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:39:58.388099] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:39:58.393127] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:01.783927] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:01.783967] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:01.784006] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:01.784029] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:01.784473] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:01.789616] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:01.789651] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:01.807316] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-12 20:40:01.816388] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-12 20:40:01.835361] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:06.179182] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:06.179223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:06.179262] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:06.179285] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:06.179727] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:06.184708] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:06.184773] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:06.190585] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:09.489136] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:09.489176] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:09.489226] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:09.489250] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:09.489696] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:09.505899] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:09.505937] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:09.515191] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:12.479103] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:12.479143] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:12.479183] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:12.479206] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:12.479649] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:12.484712] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:12.484777] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:12.509271] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-12 20:40:12.518420] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-12 20:40:12.544716] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:15.507602] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:15.507644] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:15.507684] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:15.507708] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:15.508168] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:15.513078] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:15.513115] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:15.518166] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:18.478527] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:18.478584] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:18.478627] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:18.478651] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:18.479119] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:18.483972] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:18.484008] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:18.489147] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:21.598546] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:21.598582] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:21.598619] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:21.598640] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:21.599074] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:21.603721] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:21.603782] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:21.619706] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-12 20:40:21.628273] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-12 20:40:21.652842] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:24.576064] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:24.576090] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:24.576114] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:24.576127] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:24.576408] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:24.579350] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:24.579371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:24.582465] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:27.871951] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:27.871991] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:27.872031] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:27.872055] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:27.872515] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:27.882878] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:27.882916] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:27.892342] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:31.036198] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:31.036223] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:31.036246] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:31.036259] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:31.036522] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:31.039417] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:31.039443] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:31.049110] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-12 20:40:31.054337] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-12 20:40:31.069178] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:34.122825] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:34.122850] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:34.122872] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:34.122885] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:34.123134] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:34.126090] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:34.126110] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:34.129155] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:37.073972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:37.074012] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:37.074052] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:37.074075] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:37.074519] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:37.079472] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:37.079508] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:37.084525] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:40.088485] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:40.088539] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:40.088582] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:40.088606] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:40.089072] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:40.097388] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:40.097427] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-1 to master-replicate-2 >[2012-04-12 20:40:40.116789] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-12 20:40:40.125993] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-12 20:40:40.140520] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:43.258311] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:43.258350] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:43.258389] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:43.258412] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:43.258888] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:43.263981] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:43.264027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:43.269149] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:46.648289] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:46.648331] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:46.648372] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:46.648395] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:46.648894] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:46.653884] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:46.653935] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:46.659059] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:50.266066] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:50.266109] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:50.266150] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:50.266173] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:50.266618] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:50.271592] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:50.271629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:50.276773] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:53.374956] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 69 >[2012-04-12 20:40:53.899955] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:53.899994] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:40:53.900035] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:53.900058] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:40:53.900486] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:40:53.909171] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:53.909222] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:40:53.927456] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-12 20:40:56.412977] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 73 >[2012-04-12 20:40:57.030706] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:57.030775] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:40:57.030822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:40:57.030846] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:40:57.031277] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:40:57.036334] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:40:57.036370] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-12 20:40:57.041446] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-12 20:41:00.166482] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:00.166508] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:41:00.166533] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:00.166546] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:41:00.166819] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:41:00.169701] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:00.169722] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-12 20:41:00.172805] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-1 to master-replicate-0 >[2012-04-12 20:41:03.137282] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-3: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:03.137308] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-3: No such file or directory >[2012-04-12 20:41:03.137331] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-2: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:03.137344] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-1: xattrop failed on child master-client-2: No such file or directory >[2012-04-12 20:41:03.137597] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-1 (No such file or directory) >[2012-04-12 20:41:03.140354] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:03.140375] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-1 to master-replicate-0 >[2012-04-12 20:41:03.155087] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-12 20:41:03.161919] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-12 20:41:03.173531] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:06.268522] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:06.268561] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:06.268601] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:06.268624] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:06.269085] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:06.274025] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:06.274061] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:06.279236] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:09.689273] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:09.689313] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:09.689353] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:09.689376] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:09.689850] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:09.694786] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:09.694822] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:09.699869] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:12.856365] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:12.856404] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:12.856443] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:12.856465] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:12.856927] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:12.861916] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:12.861952] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:12.886852] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-12 20:41:12.896422] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-12 20:41:12.915913] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:16.078884] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:16.078926] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:16.078967] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:16.078990] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:16.079417] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:16.084419] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:16.084470] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:16.089568] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:19.074657] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:19.074697] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:19.074767] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:19.074798] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:19.075228] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:19.080307] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:19.080344] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:19.085441] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:22.389098] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:22.389140] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:41:22.389181] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:41:22.389204] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:41:22.389666] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:41:22.394801] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:41:22.394838] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-0 to master-replicate-2 >[2012-04-12 20:41:22.418656] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 105 >[2012-04-12 20:41:22.419154] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-12 20:44:54.256571] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-12 20:44:59.258637] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-12 20:44:59.261066] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-12 20:44:59.358724] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '5047ba1e-f123-4655-bc23-cc30c9523942' >[2012-04-12 20:44:59.358803] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5' >[2012-04-12 20:44:59.358827] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-12 20:44:59.358846] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-12 20:44:59.358865] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-12 20:44:59.359102] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-12 20:44:59.359214] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-12 20:44:59.359392] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-replicate-2 >[2012-04-12 20:44:59.423140] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-12 20:44:59.428137] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-12 20:44:59.432870] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-12 20:44:59.437415] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >[2012-04-12 20:44:59.441980] I [client.c:2151:notify] 0-master-client-4: parent translators are ready, attempting connect on transport >[2012-04-12 20:44:59.446561] I [client.c:2151:notify] 0-master-client-5: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 7: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 16: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 25: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 34: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 35: end-volume > 36: > 37: volume master-client-4 > 38: type protocol/client > 39: option remote-host 172.17.251.56 > 40: option remote-subvolume /exportdir/master/d5 > 41: option transport-type tcp > 42: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 43: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 44: end-volume > 45: > 46: volume master-client-5 > 47: type protocol/client > 48: option remote-host 172.17.251.56 > 49: option remote-subvolume /exportdir/master/d6 > 50: option transport-type tcp > 51: option username 6d47c2ce-36a2-4ec6-81d9-333e35056efb > 52: option password 778680bb-2fb7-46a7-961e-5093d2ade291 > 53: end-volume > 54: > 55: volume master-replicate-0 > 56: type cluster/replicate > 57: subvolumes master-client-0 master-client-1 > 58: end-volume > 59: > 60: volume master-replicate-1 > 61: type cluster/replicate > 62: subvolumes master-client-2 master-client-3 > 63: end-volume > 64: > 65: volume master-replicate-2 > 66: type cluster/replicate > 67: subvolumes master-client-4 master-client-5 > 68: end-volume > 69: > 70: volume master-dht > 71: type cluster/distribute > 72: option decommissioned-bricks master-replicate-2 > 73: subvolumes master-replicate-0 master-replicate-1 master-replicate-2 > 74: end-volume > 75: > 76: volume master-write-behind > 77: type performance/write-behind > 78: subvolumes master-dht > 79: end-volume > 80: > 81: volume master-read-ahead > 82: type performance/read-ahead > 83: subvolumes master-write-behind > 84: end-volume > 85: > 86: volume master-io-cache > 87: type performance/io-cache > 88: subvolumes master-read-ahead > 89: end-volume > 90: > 91: volume master-quick-read > 92: type performance/quick-read > 93: subvolumes master-io-cache > 94: end-volume > 95: > 96: volume master-md-cache > 97: type performance/md-cache > 98: subvolumes master-quick-read > 99: end-volume >100: >101: volume master >102: type debug/io-stats >103: option latency-measurement off >104: option count-fop-hits off >105: subvolumes master-md-cache >106: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-12 20:44:59.453375] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-12 20:44:59.453540] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-12 20:44:59.453686] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-12 20:44:59.453864] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-12 20:44:59.454009] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-4: changing port to 24017 (from 0) >[2012-04-12 20:44:59.454150] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-5: changing port to 24018 (from 0) >[2012-04-12 20:44:59.454320] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-12 20:44:59.454377] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-12 20:44:59.454419] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-12 20:44:59.454458] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-12 20:44:59.454560] I [client.c:136:client_register_grace_timer] 0-master-client-4: Registering a grace timer >[2012-04-12 20:44:59.454606] I [client.c:136:client_register_grace_timer] 0-master-client-5: Registering a grace timer >[2012-04-12 20:45:03.275643] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-12 20:45:03.275878] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.280834] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-12 20:45:03.280999] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.286140] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-12 20:45:03.286305] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.291209] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-12 20:45:03.291389] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.295504] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-12 20:45:03.295540] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.295764] I [afr-common.c:3535:afr_notify] 0-master-replicate-1: Subvolume 'master-client-3' came back up; going online. >[2012-04-12 20:45:03.295836] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-12 20:45:03.295861] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.296053] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-12 20:45:03.296084] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.296169] I [afr-common.c:3535:afr_notify] 0-master-replicate-0: Subvolume 'master-client-1' came back up; going online. >[2012-04-12 20:45:03.296228] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-12 20:45:03.296253] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.297806] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-12 20:45:03.297859] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-12 20:45:03.297897] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-12 20:45:03.297932] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-12 20:45:03.299777] W [client.c:2078:client_rpc_notify] 0-master-client-4: Cancelling the grace timer >[2012-04-12 20:45:03.299941] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-4: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.300306] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-4: Connected to 172.17.251.56:24017, attached to remote volume '/exportdir/master/d5'. >[2012-04-12 20:45:03.300351] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.300444] I [afr-common.c:3535:afr_notify] 0-master-replicate-2: Subvolume 'master-client-4' came back up; going online. >[2012-04-12 20:45:03.300628] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-4: Server lk version = 1 >[2012-04-12 20:45:03.305411] W [client.c:2078:client_rpc_notify] 0-master-client-5: Cancelling the grace timer >[2012-04-12 20:45:03.305576] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-5: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 20:45:03.305975] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-5: Connected to 172.17.251.56:24018, attached to remote volume '/exportdir/master/d6'. >[2012-04-12 20:45:03.306008] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 20:45:03.306863] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-5: Server lk version = 1 >[2012-04-12 20:45:03.334275] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-0: added root inode >[2012-04-12 20:45:03.337545] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-1: added root inode >[2012-04-12 20:45:03.337689] I [afr-common.c:1877:afr_set_root_inode_on_first_lookup] 0-master-replicate-2: added root inode >[2012-04-12 20:45:03.337973] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-12 20:45:03.346607] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-12 20:45:03.365590] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-12 20:45:03.375277] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-12 20:45:03.402009] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:07.453222] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:07.453274] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:07.453335] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:07.453387] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:07.453956] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:07.458949] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:07.458984] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:07.462322] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:11.692414] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:11.692453] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:11.692491] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:11.692514] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:11.692981] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:11.697950] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:11.697986] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:11.702949] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:14.788337] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:14.788389] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:14.788430] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:14.788453] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:14.788915] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:14.793890] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:14.793926] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:14.805547] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-12 20:45:14.814636] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-12 20:45:14.840993] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:17.820679] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:17.820718] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:17.820790] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:17.820817] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:17.821242] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:17.826203] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:17.826239] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:17.835407] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:20.821801] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:20.821841] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:20.821880] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:20.821915] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:20.822361] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:20.827335] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:20.827371] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:20.836908] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:23.823500] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:23.823538] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:23.823577] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:23.823600] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:23.824056] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:23.829009] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:23.829045] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:45:23.840710] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-12 20:45:23.849813] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-12 20:45:23.876431] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:26.968516] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:26.968556] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:26.977098] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:26.977150] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:26.977655] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:26.993672] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:26.993728] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:27.010275] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:29.839283] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:29.839323] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:29.839395] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:29.839421] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:29.839879] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:29.844812] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:29.844849] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:29.854113] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:32.827691] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:32.827730] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:32.827808] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:32.827831] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:32.828258] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:32.833209] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:32.833244] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:32.844954] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-12 20:45:32.854023] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-12 20:45:32.880758] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:35.921384] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:35.921423] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:35.921462] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:35.921492] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:35.922425] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:35.938923] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:35.938966] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:35.955258] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:39.083448] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:39.083506] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:39.083548] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:39.083571] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:39.084031] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:39.089056] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:39.089092] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:39.098765] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:42.395767] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:42.395808] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:42.395850] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:42.395872] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:42.396315] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:42.404852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:42.404890] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:42.427159] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-12 20:45:42.439253] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-12 20:45:42.465991] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:45.686835] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:45.686861] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:45.686883] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:45.686896] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:45.687149] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:45.690190] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:45.690211] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:45.696103] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:48.861972] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:48.862014] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:48.862054] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:48.862076] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:48.862517] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:48.870169] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:48.870205] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:48.879810] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:52.309822] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:52.309877] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:45:52.309926] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:52.309950] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:45:52.310397] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:45:52.315485] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:52.315521] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-replicate-2 to master-replicate-1 >[2012-04-12 20:45:52.327398] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-12 20:45:52.336554] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-12 20:45:52.351134] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:45:55.508373] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:55.508414] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:45:55.508652] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:55.508685] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:45:55.509147] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:45:55.533135] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:55.533172] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:45:55.549380] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:45:56.445287] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 52 >[2012-04-12 20:45:58.425216] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 52 >[2012-04-12 20:45:58.973727] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:58.973797] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:45:58.973839] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:45:58.973862] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:45:58.974290] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:45:58.979342] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:45:58.979379] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:45:58.989100] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-replicate-0 to master-replicate-1 >[2012-04-12 20:46:00.328311] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 53 >[2012-04-12 20:46:01.902410] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 53 >[2012-04-12 20:46:02.223591] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-1: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:02.223628] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-1: No such file or directory >[2012-04-12 20:46:02.223666] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-0: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:02.223689] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-0: xattrop failed on child master-client-0: No such file or directory >[2012-04-12 20:46:02.224144] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-replicate-0 (No such file or directory) >[2012-04-12 20:46:02.233249] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:02.233300] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-replicate-0 to master-replicate-1 >[2012-04-12 20:46:02.269474] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:05.275793] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:05.275836] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:05.276526] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:05.276561] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:05.277058] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:05.284349] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:05.284387] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:05.301663] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:08.238831] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:08.238870] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:08.238909] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:08.238931] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:08.239373] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:08.244399] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:08.244435] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:08.253999] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:09.826882] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 62 >[2012-04-12 20:46:11.506982] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:11.507020] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:11.507059] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:11.507081] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:11.507524] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:11.514084] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:11.514122] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:11.524271] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:11.948469] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 63 >[2012-04-12 20:46:13.699874] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 63 >[2012-04-12 20:46:14.715562] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:14.715586] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:14.715618] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:14.715632] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:14.715902] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:14.718817] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:14.718839] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:14.725571] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-12 20:46:14.730794] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-12 20:46:14.756481] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:15.669232] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 71 >[2012-04-12 20:46:17.463769] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 71 >[2012-04-12 20:46:18.132200] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:18.132227] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:18.132253] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:18.132269] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:18.132572] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:18.135936] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:18.135960] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:18.142251] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:19.093861] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 72 >[2012-04-12 20:46:20.907469] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 72 >[2012-04-12 20:46:21.495123] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:21.495160] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:21.495199] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:21.495221] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:21.495662] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:21.500627] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:21.500675] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:21.510001] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:24.551678] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:24.551717] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:24.551789] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:24.551817] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:24.552244] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:24.557261] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:24.557311] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:24.569065] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-12 20:46:24.578248] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-12 20:46:24.605120] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:27.615048] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:27.615074] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:27.615097] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:27.615110] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:27.615380] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:27.618312] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:27.618334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:27.624909] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:30.632502] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:30.632529] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:30.632552] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:30.632565] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:30.632835] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:30.635608] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:30.635629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:30.644212] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:34.118193] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-5: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:34.118232] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-5: No such file or directory >[2012-04-12 20:46:34.118271] W [client3_1-fops.c:1704:client3_1_xattrop_cbk] 0-master-client-4: remote operation failed: No such file or directory. Path: (null) >[2012-04-12 20:46:34.118294] E [afr-transaction.c:756:afr_changelog_pre_op_cbk] 0-master-replicate-2: xattrop failed on child master-client-4: No such file or directory >[2012-04-12 20:46:34.118767] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-replicate-2 (No such file or directory) >[2012-04-12 20:46:34.123812] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 20:46:34.123848] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-replicate-2 to master-replicate-0 >[2012-04-12 20:46:34.135246] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 83 >[2012-04-12 20:46:34.148565] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-12 23:19:42.469273] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-12 23:19:47.482678] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-12 23:19:47.482790] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-12 23:19:47.485636] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '9d5940f1-a0b4-4fee-910c-d8cba200af43' >[2012-04-12 23:19:47.485682] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '1' >[2012-04-12 23:19:47.485717] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-12 23:19:47.487319] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-12 23:19:47.487366] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-12 23:19:47.487695] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-12 23:19:47.487918] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-12 23:19:47.493574] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-12 23:19:47.498327] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-12 23:19:47.503086] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-12 23:19:47.507707] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 7: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 16: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 25: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 34: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 35: end-volume > 36: > 37: volume master-dht > 38: type cluster/distribute > 39: subvolumes master-client-0 master-client-1 master-client-2 master-client-3 > 40: end-volume > 41: > 42: volume master-write-behind > 43: type performance/write-behind > 44: subvolumes master-dht > 45: end-volume > 46: > 47: volume master-read-ahead > 48: type performance/read-ahead > 49: subvolumes master-write-behind > 50: end-volume > 51: > 52: volume master-io-cache > 53: type performance/io-cache > 54: subvolumes master-read-ahead > 55: end-volume > 56: > 57: volume master-quick-read > 58: type performance/quick-read > 59: subvolumes master-io-cache > 60: end-volume > 61: > 62: volume master-md-cache > 63: type performance/md-cache > 64: subvolumes master-quick-read > 65: end-volume > 66: > 67: volume master > 68: type debug/io-stats > 69: option latency-measurement off > 70: option count-fop-hits off > 71: subvolumes master-md-cache > 72: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-12 23:19:47.518328] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-12 23:19:47.518505] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-12 23:19:47.518653] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-12 23:19:47.518834] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-12 23:19:47.518988] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-12 23:19:47.519044] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-12 23:19:47.519086] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-12 23:19:47.519124] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-12 23:19:51.499621] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-12 23:19:51.499857] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 23:19:51.500290] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-12 23:19:51.500332] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 23:19:51.501078] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-12 23:19:51.505923] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-12 23:19:51.506087] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 23:19:51.506444] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-12 23:19:51.506492] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 23:19:51.506987] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-12 23:19:51.511801] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-12 23:19:51.511965] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 23:19:51.512320] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-12 23:19:51.512353] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 23:19:51.512851] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-12 23:19:51.517719] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-12 23:19:51.517917] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-12 23:19:51.518275] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-12 23:19:51.518308] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-12 23:19:51.518883] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-12 23:19:51.531871] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-12 23:19:51.535557] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-12 23:19:51.550146] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-12 23:19:51.554874] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-12 23:19:51.579011] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file5: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:19:54.110931] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:19:54.110989] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file5: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:19:54.113417] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:19:54.113454] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file5 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:19:54.115399] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file6: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:19:56.426239] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:19:56.426322] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file6: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:19:56.429274] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:19:56.429332] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file6 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:19:56.440396] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file8: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:19:58.288080] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:19:58.288169] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file8: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:19:58.290496] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:19:58.290530] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file8 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:19:58.300834] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file7: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:00.411185] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:00.411225] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file7: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:00.412766] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:00.412789] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file7 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:00.421956] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-12 23:20:00.424450] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-12 23:20:00.431001] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file5: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:02.079188] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:02.079246] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file5: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:02.081704] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:02.081772] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file5 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:02.085254] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file6: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:03.523787] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:03.523848] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file6: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:03.526273] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:03.526310] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file6 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:03.529834] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file8: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:04.975146] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:04.975216] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file8: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:04.977636] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:04.977672] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file8 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:04.984195] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file7: attempting to move from master-client-1 to master-client-2 >[2012-04-12 23:20:06.855169] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:06.855207] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file7: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:06.856666] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:06.856688] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file7 from subvolume master-client-1 to master-client-2 >[2012-04-12 23:20:06.865276] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-12 23:20:06.867592] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-12 23:20:06.873876] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file5: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:08.819055] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:08.819114] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file5: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:08.821627] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:08.821664] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file5 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:08.825200] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file6: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:10.745116] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:10.745154] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file6: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:10.746461] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:10.746482] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file6 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:10.748548] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file8: attempting to move from master-client-0 to master-client-3 >[2012-04-12 23:20:12.144115] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:12.144151] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file8: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:12.145634] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:12.145655] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file8 from subvolume master-client-0 to master-client-3 >[2012-04-12 23:20:12.150173] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file7: attempting to move from master-client-1 to master-client-2 >[2012-04-12 23:20:13.578887] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:13.578946] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file7: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:13.582820] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:13.582857] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file7 from subvolume master-client-1 to master-client-2 >[2012-04-12 23:20:13.597882] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-12 23:20:13.602270] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-12 23:20:13.613676] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file7: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:15.395980] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:15.396040] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file7: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:15.398445] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:15.398482] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file7 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:15.406829] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file5: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:17.159310] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:17.159384] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file5: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:17.161804] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:17.161841] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file5 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:17.165312] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file6: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:18.744191] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:18.744230] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file6: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:18.745659] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:18.745680] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file6 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:18.747822] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file8: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:19.862278] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:19.862316] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file8: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:19.863754] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:19.863776] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file8 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:19.874613] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-12 23:20:19.876936] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-12 23:20:19.882242] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file7: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:21.160237] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:21.160285] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file7: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:21.162585] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:21.162614] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file7 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:21.169903] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file5: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:21.391109] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 64 >[2012-04-12 23:20:22.298206] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:22.298245] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file5: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:22.302059] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:22.302081] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file5 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:22.304192] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file6: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:23.325472] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 65 >[2012-04-12 23:20:23.370436] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:23.370470] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file6: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:23.371956] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:23.371978] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file6 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:23.374165] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file8: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:24.607866] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:24.607924] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file8: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:24.610359] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:24.610395] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file8 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:24.624842] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-12 23:20:24.629066] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-12 23:20:24.638297] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file7: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:25.533425] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 73 >[2012-04-12 23:20:26.087571] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:26.087608] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file7: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:26.089058] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:26.089079] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file7 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:26.093947] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file5: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:28.044091] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:28.044148] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file5: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:28.046564] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:28.046600] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file5 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:28.050177] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file6: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:29.368453] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:29.368503] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file6: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:29.370042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:29.370063] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file6 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:29.372173] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file8: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:30.496811] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:30.496854] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file8: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:30.498613] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:30.498638] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file8 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:30.508833] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-12 23:20:30.511653] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-12 23:20:30.518878] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file5: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:31.700983] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:31.701027] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file5: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:31.702796] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:31.702821] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file5 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:31.706884] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file6: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:33.218043] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:33.218081] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file6: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:33.219588] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:33.219610] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file6 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:33.221816] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file8: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:34.387094] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:34.387130] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file8: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:34.388782] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:34.388804] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file8 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:34.392991] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file7: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:35.817452] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:35.817491] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file7: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:35.819005] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:35.819027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file7 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:35.827619] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-12 23:20:35.829973] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-12 23:20:35.835316] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file7: attempting to move from master-client-0 to master-client-2 >[2012-04-12 23:20:37.287588] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-12 23:20:37.287625] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file7: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-12 23:20:37.289100] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file7: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:37.289122] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file7 from subvolume master-client-0 to master-client-2 >[2012-04-12 23:20:37.294014] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file5: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:38.553869] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:38.553938] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file5: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:38.556369] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file5: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:38.556406] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file5 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:38.559940] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file6: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:40.126751] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:40.126792] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file6: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:40.128197] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file6: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:40.128218] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file6 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:40.130512] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file8: attempting to move from master-client-1 to master-client-3 >[2012-04-12 23:20:41.256249] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-12 23:20:41.256293] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file8: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-12 23:20:41.257825] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file8: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-12 23:20:41.257847] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file8 from subvolume master-client-1 to master-client-3 >[2012-04-12 23:20:41.267841] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 32, size: 3200000000, lookups: 112 >[2012-04-12 23:20:41.268153] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down >[2012-04-13 00:06:20.438274] I [glusterfsd.c:1628:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3git >[2012-04-13 00:06:25.450213] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 >[2012-04-13 00:06:25.450948] W [socket.c:1885:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported >[2012-04-13 00:06:25.530734] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'node-uuid' for volume 'master-dht' with value '9d5940f1-a0b4-4fee-910c-d8cba200af43' >[2012-04-13 00:06:25.530823] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'rebalance-cmd' for volume 'master-dht' with value '5' >[2012-04-13 00:06:25.530849] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'assert-no-child-down' for volume 'master-dht' with value 'yes' >[2012-04-13 00:06:25.530868] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'lookup-unhashed' for volume 'master-dht' with value 'yes' >[2012-04-13 00:06:25.530887] I [graph.c:250:gf_add_cmdline_options] 0-master-dht: adding option 'use-readdirp' for volume 'master-dht' with value 'yes' >[2012-04-13 00:06:25.531075] I [io-cache.c:1558:check_cache_size_ok] 0-master-quick-read: Max cache size is 2104397824 >[2012-04-13 00:06:25.531189] I [io-cache.c:1558:check_cache_size_ok] 0-master-io-cache: Max cache size is 2104397824 >[2012-04-13 00:06:25.531354] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-client-0 >[2012-04-13 00:06:25.531413] I [dht.c:278:dht_parse_decommissioned_bricks] 0-master-dht: decommissioning subvolume master-client-1 >[2012-04-13 00:06:25.536078] I [client.c:2151:notify] 0-master-client-0: parent translators are ready, attempting connect on transport >[2012-04-13 00:06:25.540894] I [client.c:2151:notify] 0-master-client-1: parent translators are ready, attempting connect on transport >[2012-04-13 00:06:25.545450] I [client.c:2151:notify] 0-master-client-2: parent translators are ready, attempting connect on transport >[2012-04-13 00:06:25.550064] I [client.c:2151:notify] 0-master-client-3: parent translators are ready, attempting connect on transport >Given volfile: >+------------------------------------------------------------------------------+ > 1: volume master-client-0 > 2: type protocol/client > 3: option remote-host 172.17.251.56 > 4: option remote-subvolume /exportdir/master/d1 > 5: option transport-type tcp > 6: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 7: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 8: end-volume > 9: > 10: volume master-client-1 > 11: type protocol/client > 12: option remote-host 172.17.251.56 > 13: option remote-subvolume /exportdir/master/d2 > 14: option transport-type tcp > 15: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 16: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 17: end-volume > 18: > 19: volume master-client-2 > 20: type protocol/client > 21: option remote-host 172.17.251.56 > 22: option remote-subvolume /exportdir/master/d3 > 23: option transport-type tcp > 24: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 25: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 26: end-volume > 27: > 28: volume master-client-3 > 29: type protocol/client > 30: option remote-host 172.17.251.56 > 31: option remote-subvolume /exportdir/master/d4 > 32: option transport-type tcp > 33: option username b0def7c2-ba7f-48eb-916b-26d7e51e30fb > 34: option password 0554411e-3acf-4f7e-a4ac-cf04c9f1edd6 > 35: end-volume > 36: > 37: volume master-dht > 38: type cluster/distribute > 39: option decommissioned-bricks master-client-0,master-client-1 > 40: subvolumes master-client-0 master-client-1 master-client-2 master-client-3 > 41: end-volume > 42: > 43: volume master-write-behind > 44: type performance/write-behind > 45: subvolumes master-dht > 46: end-volume > 47: > 48: volume master-read-ahead > 49: type performance/read-ahead > 50: subvolumes master-write-behind > 51: end-volume > 52: > 53: volume master-io-cache > 54: type performance/io-cache > 55: subvolumes master-read-ahead > 56: end-volume > 57: > 58: volume master-quick-read > 59: type performance/quick-read > 60: subvolumes master-io-cache > 61: end-volume > 62: > 63: volume master-md-cache > 64: type performance/md-cache > 65: subvolumes master-quick-read > 66: end-volume > 67: > 68: volume master > 69: type debug/io-stats > 70: option latency-measurement off > 71: option count-fop-hits off > 72: subvolumes master-md-cache > 73: end-volume > >+------------------------------------------------------------------------------+ >[2012-04-13 00:06:25.556121] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-0: changing port to 24009 (from 0) >[2012-04-13 00:06:25.556284] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-1: changing port to 24010 (from 0) >[2012-04-13 00:06:25.556448] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-2: changing port to 24011 (from 0) >[2012-04-13 00:06:25.556592] I [rpc-clnt.c:1669:rpc_clnt_reconfig] 0-master-client-3: changing port to 24012 (from 0) >[2012-04-13 00:06:25.556795] I [client.c:136:client_register_grace_timer] 0-master-client-0: Registering a grace timer >[2012-04-13 00:06:25.556884] I [client.c:136:client_register_grace_timer] 0-master-client-1: Registering a grace timer >[2012-04-13 00:06:25.556933] I [client.c:136:client_register_grace_timer] 0-master-client-2: Registering a grace timer >[2012-04-13 00:06:25.556985] I [client.c:136:client_register_grace_timer] 0-master-client-3: Registering a grace timer >[2012-04-13 00:06:29.468618] W [client.c:2078:client_rpc_notify] 0-master-client-0: Cancelling the grace timer >[2012-04-13 00:06:29.471047] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-0: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-13 00:06:29.474010] W [client.c:2078:client_rpc_notify] 0-master-client-1: Cancelling the grace timer >[2012-04-13 00:06:29.474178] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-1: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-13 00:06:29.479491] W [client.c:2078:client_rpc_notify] 0-master-client-2: Cancelling the grace timer >[2012-04-13 00:06:29.479662] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-2: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-13 00:06:29.484827] W [client.c:2078:client_rpc_notify] 0-master-client-3: Cancelling the grace timer >[2012-04-13 00:06:29.484998] I [client-handshake.c:1632:select_server_supported_programs] 0-master-client-3: Using Program GlusterFS 3git, Num (1298437), Version (330) >[2012-04-13 00:06:29.489363] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-0: Connected to 172.17.251.56:24009, attached to remote volume '/exportdir/master/d1'. >[2012-04-13 00:06:29.489412] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-13 00:06:29.489733] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-1: Connected to 172.17.251.56:24010, attached to remote volume '/exportdir/master/d2'. >[2012-04-13 00:06:29.489798] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-13 00:06:29.489929] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-2: Connected to 172.17.251.56:24011, attached to remote volume '/exportdir/master/d3'. >[2012-04-13 00:06:29.489958] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-13 00:06:29.490082] I [client-handshake.c:1429:client_setvolume_cbk] 0-master-client-3: Connected to 172.17.251.56:24012, attached to remote volume '/exportdir/master/d4'. >[2012-04-13 00:06:29.490111] I [client-handshake.c:1441:client_setvolume_cbk] 0-master-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2012-04-13 00:06:29.491776] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-2: Server lk version = 1 >[2012-04-13 00:06:29.491831] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-1: Server lk version = 1 >[2012-04-13 00:06:29.491868] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-0: Server lk version = 1 >[2012-04-13 00:06:29.491902] I [client-handshake.c:456:client_set_lk_version_cbk] 0-master-client-3: Server lk version = 1 >[2012-04-13 00:06:29.505006] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of / >[2012-04-13 00:06:29.508661] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on / >[2012-04-13 00:06:29.522890] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large1 >[2012-04-13 00:06:29.527382] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large1 >[2012-04-13 00:06:29.533610] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file0: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:06:31.701142] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:31.701202] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file0: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:31.703611] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:31.703647] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file0 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:06:31.707593] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file1: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:06:33.312177] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:33.312215] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file1: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:33.313651] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:33.313672] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file1 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:06:33.317229] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file2: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:06:34.670151] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:34.670210] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file2: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:34.672628] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:34.672665] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file2 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:06:34.679757] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file3: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:06:36.092096] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:36.092153] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file3: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:36.094982] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:36.095019] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file3 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:06:36.101007] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file4: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:06:37.445182] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:37.445220] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file4: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:37.448127] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:37.448149] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file4 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:06:37.451829] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large1/file9: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:06:38.767356] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:38.767414] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large1/file9: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:38.769857] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large1/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:38.769894] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large1/file9 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:06:38.790623] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large2 >[2012-04-13 00:06:38.794974] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large2 >[2012-04-13 00:06:38.802711] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file0: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:40.269666] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 6, size: 600000000, lookups: 17 >[2012-04-13 00:06:40.673495] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:40.673553] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file0: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:40.675990] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:40.676027] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file0 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:40.679683] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file1: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:42.032099] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 700000000, lookups: 18 >[2012-04-13 00:06:42.218490] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:42.218527] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file1: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:42.220010] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:42.220033] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file1 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:42.222605] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file2: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:43.544401] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:43.544460] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file2: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:43.546949] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:43.546987] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file2 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:43.551714] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file3: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:43.560925] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 900000000, lookups: 20 >[2012-04-13 00:06:44.842580] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 900000000, lookups: 20 >[2012-04-13 00:06:45.677568] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:45.677625] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file3: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:45.679966] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:45.680011] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file3 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:45.684084] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file4: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:46.691963] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 10, size: 1000000000, lookups: 21 >[2012-04-13 00:06:47.222276] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 10, size: 1000000000, lookups: 21 >[2012-04-13 00:06:47.812602] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:47.812658] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file4: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:47.815045] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:47.815081] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file4 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:47.818713] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large2/file9: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:49.190537] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 11, size: 1100000000, lookups: 22 >[2012-04-13 00:06:49.373156] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:49.373191] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large2/file9: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:49.374608] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large2/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:49.374629] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large2/file9 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:49.385681] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large3 >[2012-04-13 00:06:49.388026] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large3 >[2012-04-13 00:06:49.392249] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file0: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:50.714130] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 12, size: 1200000000, lookups: 33 >[2012-04-13 00:06:50.923195] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:50.923236] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file0: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:50.924680] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:50.924701] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file0 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:50.927077] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file1: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:52.234214] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 13, size: 1300000000, lookups: 34 >[2012-04-13 00:06:52.468341] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:52.468388] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file1: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:52.469828] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:52.469849] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file1 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:52.472201] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file2: attempting to move from master-client-0 to master-client-3 >[2012-04-13 00:06:53.811692] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 1400000000, lookups: 35 >[2012-04-13 00:06:53.969003] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:06:53.969040] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file2: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:06:53.970513] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:53.970535] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file2 from subvolume master-client-0 to master-client-3 >[2012-04-13 00:06:53.973957] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file3: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:55.378017] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 15, size: 1500000000, lookups: 36 >[2012-04-13 00:06:55.550447] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:55.550505] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file3: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:55.552937] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:55.552975] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file3 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:55.556632] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file4: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:56.813821] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 1600000000, lookups: 37 >[2012-04-13 00:06:57.046289] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:57.046347] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file4: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:57.048852] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:57.048890] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file4 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:57.053151] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large3/file9: attempting to move from master-client-1 to master-client-2 >[2012-04-13 00:06:58.557539] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 1700000000, lookups: 38 >[2012-04-13 00:06:58.707058] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:06:58.707094] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large3/file9: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:06:58.708580] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large3/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:06:58.708602] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large3/file9 from subvolume master-client-1 to master-client-2 >[2012-04-13 00:06:58.719945] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large4 >[2012-04-13 00:06:58.722375] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large4 >[2012-04-13 00:06:58.727021] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file3: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:00.326570] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 18, size: 1800000000, lookups: 49 >[2012-04-13 00:07:00.420848] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:00.420884] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file3: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:00.422312] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:00.422334] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file3 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:00.424782] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file4: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:01.609883] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 19, size: 1900000000, lookups: 50 >[2012-04-13 00:07:01.899096] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:01.899149] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file4: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:01.901393] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:01.901427] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file4 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:01.904949] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file9: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:03.306337] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 2000000000, lookups: 51 >[2012-04-13 00:07:03.444197] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:03.444233] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file9: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:03.445642] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:03.445663] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file9 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:03.448698] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file0: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:04.784264] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 21, size: 2100000000, lookups: 52 >[2012-04-13 00:07:04.945294] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:04.945329] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file0: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:04.946781] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:04.946803] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file0 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:04.949265] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file1: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:05.849131] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 2200000000, lookups: 53 >[2012-04-13 00:07:06.259421] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:06.259457] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file1: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:06.260890] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:06.260911] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file1 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:06.263339] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large4/file2: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:07.348730] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 23, size: 2300000000, lookups: 54 >[2012-04-13 00:07:07.664915] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:07.664972] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large4/file2: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:07.667350] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large4/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:07.667387] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large4/file2 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:07.695387] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large5 >[2012-04-13 00:07:07.699706] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large5 >[2012-04-13 00:07:07.710098] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file3: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:09.373139] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 24, size: 2400000000, lookups: 65 >[2012-04-13 00:07:09.507392] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:09.507448] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file3: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:09.511316] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:09.511353] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file3 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:09.515571] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file4: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:10.943960] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 25, size: 2500000000, lookups: 66 >[2012-04-13 00:07:11.104956] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:11.105007] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file4: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:11.107175] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:11.107207] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file4 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:11.111133] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file9: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:12.579725] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 26, size: 2600000000, lookups: 67 >[2012-04-13 00:07:13.129100] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:13.129157] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file9: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:13.131836] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:13.131874] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file9 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:13.136773] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file0: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:14.300458] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 27, size: 2700000000, lookups: 68 >[2012-04-13 00:07:14.643903] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:14.643953] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file0: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:14.646150] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:14.646184] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file0 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:14.649563] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file1: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:15.948362] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 28, size: 2800000000, lookups: 69 >[2012-04-13 00:07:16.113556] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:16.113599] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file1: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:16.115038] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:16.115060] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file1 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:16.117261] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large5/file2: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:17.212795] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 29, size: 2900000000, lookups: 70 >[2012-04-13 00:07:17.342779] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:17.342813] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large5/file2: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:17.344200] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large5/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:17.344222] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large5/file2 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:17.355682] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large6 >[2012-04-13 00:07:17.358103] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large6 >[2012-04-13 00:07:17.365385] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file3: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:19.016066] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:19.016122] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file3: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:19.018497] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:19.018534] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file3 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:19.022425] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file4: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:20.369374] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 31, size: 3100000000, lookups: 82 >[2012-04-13 00:07:20.754000] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:20.754057] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file4: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:20.756480] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:20.756516] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file4 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:20.759974] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file9: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:21.949229] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 32, size: 3200000000, lookups: 83 >[2012-04-13 00:07:22.070663] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:22.070703] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file9: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:22.072215] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:22.072238] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file9 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:22.076231] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file0: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:23.120173] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:23.120248] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file0: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:23.124028] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:23.124066] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file0 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:23.127627] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file1: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:23.908359] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 34, size: 3400000000, lookups: 85 >[2012-04-13 00:07:24.501183] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:24.501218] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file1: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:24.502543] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:24.502564] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file1 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:24.505653] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large6/file2: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:25.585177] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 35, size: 3500000000, lookups: 86 >[2012-04-13 00:07:25.702059] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:25.702092] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large6/file2: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:25.703471] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large6/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:25.703493] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large6/file2 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:25.714524] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large7 >[2012-04-13 00:07:25.716901] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large7 >[2012-04-13 00:07:25.721195] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file0: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:26.956374] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:26.956419] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file0: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:26.958122] W [dht-common.c:945:dht_lookup_everywhere_cbk] 0-master-dht: multiple subvolumes (master-client-3 and master-client-2) have file /large7/file0 (preferably rename the file in the backend, and do a fresh lookup) >[2012-04-13 00:07:26.958196] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:26.958224] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file0 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:26.959883] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file1 on master-client-3 >[2012-04-13 00:07:26.963206] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file1: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:28.242774] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:28.242832] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file1: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:28.245162] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:28.245200] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file1 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:28.247918] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file2 on master-client-3 >[2012-04-13 00:07:28.252473] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file2: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:29.536649] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:29.536684] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file2: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:29.538088] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:29.538109] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file2 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:29.540189] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file3 on master-client-2 >[2012-04-13 00:07:29.542884] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file3: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:30.817169] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:30.817226] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file3: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:30.819590] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:30.819626] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file3 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:30.822001] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file4 on master-client-2 >[2012-04-13 00:07:30.826454] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file4: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:32.329568] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:32.329602] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file4: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:32.330993] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:32.331014] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file4 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:32.332367] I [dht-common.c:959:dht_lookup_everywhere_cbk] 0-master-dht: deleting stale linkfile /large7/file9 on master-client-2 >[2012-04-13 00:07:32.334982] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large7/file9: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:33.577965] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:33.578020] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large7/file9: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:33.580391] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large7/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:33.580428] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large7/file9 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:33.602385] I [dht-common.c:2347:dht_setxattr] 0-master-dht: fixing the layout of /large8 >[2012-04-13 00:07:33.606653] I [dht-rebalance.c:1064:gf_defrag_migrate_data] 0-master-dht: migrate data called on /large8 >[2012-04-13 00:07:33.614205] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file3: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:35.177794] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:35.177847] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file3: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:35.180042] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file3: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:35.180084] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file3 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:35.183414] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file4: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:36.742035] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:36.742090] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file4: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:36.744423] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file4: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:36.744460] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file4 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:36.748106] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file9: attempting to move from master-client-0 to master-client-2 >[2012-04-13 00:07:38.222033] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-0: remote operation failed: No such file or directory >[2012-04-13 00:07:38.222090] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file9: failed to perform truncate on master-client-0 (No such file or directory) >[2012-04-13 00:07:38.224419] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file9: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:38.224455] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file9 from subvolume master-client-0 to master-client-2 >[2012-04-13 00:07:38.229198] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file0: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:39.697882] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:39.697916] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file0: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:39.699290] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file0: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:39.699311] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file0 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:39.701450] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file1: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:41.158656] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:41.158712] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file1: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:41.161105] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file1: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:41.161142] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file1 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:41.164625] I [dht-rebalance.c:647:dht_migrate_file] 0-master-dht: /large8/file2: attempting to move from master-client-1 to master-client-3 >[2012-04-13 00:07:42.554399] W [client3_1-fops.c:708:client3_1_truncate_cbk] 0-master-client-1: remote operation failed: No such file or directory >[2012-04-13 00:07:42.554457] W [dht-rebalance.c:834:dht_migrate_file] 0-master-dht: /large8/file2: failed to perform truncate on master-client-1 (No such file or directory) >[2012-04-13 00:07:42.556840] W [dht-rebalance.c:849:dht_migrate_file] 0-master-dht: /large8/file2: failed to lookup the file on subvolumes (No such file or directory) >[2012-04-13 00:07:42.556888] I [dht-rebalance.c:854:dht_migrate_file] 0-master-dht: completed migration of /large8/file2 from subvolume master-client-1 to master-client-3 >[2012-04-13 00:07:42.580249] I [dht-rebalance.c:1580:gf_defrag_status_get] 0-glusterfs: Files migrated: 48, size: 4800000000, lookups: 128 >[2012-04-13 00:07:42.580686] W [glusterfsd.c:793:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3e65ee5ccd] (-->/lib64/libpthread.so.0() [0x3e666077f1] (-->/usr/local/sbin/glusterfs(glusterfs_sigwaiter+0xfc) [0x407718]))) 0-: received signum (15), shutting down
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 812314
:
577299
| 577300