Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 841817 Details for
Bug 1046603
BVT: Rebalance failed for files with xattrs
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Rebalance logs from masternode
hosdu-rebalance.log (text/x-log), 114.55 KB, created by
Lalatendu Mohanty
on 2013-12-26 09:51:40 UTC
(
hide
)
Description:
Rebalance logs from masternode
Filename:
MIME Type:
Creator:
Lalatendu Mohanty
Created:
2013-12-26 09:51:40 UTC
Size:
114.55 KB
patch
obsolete
>[2013-12-25 07:13:10.806923] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=598cd7c0-034b-42a1-9bab-6d84e03fc60f --socket-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.pid -l /var/log/glusterfs/hosdu-rebalance.log) >[2013-12-25 07:13:10.808637] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 >[2013-12-25 07:13:10.812636] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled >[2013-12-25 07:13:10.812678] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread >[2013-12-25 07:13:10.812950] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled >[2013-12-25 07:13:10.812981] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread >[2013-12-25 07:13:15.812715] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '598cd7c0-034b-42a1-9bab-6d84e03fc60f' >[2013-12-25 07:13:15.812758] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1' >[2013-12-25 07:13:15.812778] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on' >[2013-12-25 07:13:15.812797] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:13:15.812815] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:13:15.812832] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:13:15.812850] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:13:15.812880] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:13:15.812898] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:13:15.812915] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:13:15.812933] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:13:15.812950] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:13:15.812967] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:13:15.812984] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:13:15.813015] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:13:15.813046] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:13:15.813063] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:13:15.813091] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:13:15.813111] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:13:15.813128] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:13:15.813145] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:13:15.813193] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:13:15.814037] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2013-12-25 07:13:15.858679] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled >[2013-12-25 07:13:15.858732] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread >[2013-12-25 07:13:15.862585] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled >[2013-12-25 07:13:15.862606] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread >[2013-12-25 07:13:15.866164] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled >[2013-12-25 07:13:15.866205] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread >[2013-12-25 07:13:15.870271] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled >[2013-12-25 07:13:15.870291] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread >[2013-12-25 07:13:15.874804] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled >[2013-12-25 07:13:15.874826] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread >[2013-12-25 07:13:15.878846] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled >[2013-12-25 07:13:15.878868] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread >[2013-12-25 07:13:15.882507] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled >[2013-12-25 07:13:15.882529] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread >[2013-12-25 07:13:15.886010] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled >[2013-12-25 07:13:15.886031] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread >[2013-12-25 07:13:15.886106] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:15.893695] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:15.989654] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:15.996572] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:16.003467] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:16.010297] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:16.017053] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport >[2013-12-25 07:13:16.023995] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume hosdu-client-0 > 2: type protocol/client > 3: option remote-host rhsauto019.lab.eng.blr.redhat.com > 4: option remote-subvolume /bricks/hosdu_brick0 > 5: option transport-type socket > 6: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 7: option password d94d42e0-803a-405b-8178-23078d9555b8 > 8: end-volume > 9: > 10: volume hosdu-client-1 > 11: type protocol/client > 12: option remote-host rhsauto008.lab.eng.blr.redhat.com > 13: option remote-subvolume /bricks/hosdu_brick1 > 14: option transport-type socket > 15: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 16: option password d94d42e0-803a-405b-8178-23078d9555b8 > 17: end-volume > 18: > 19: volume hosdu-replicate-0 > 20: type cluster/replicate > 21: option data-self-heal off > 22: option metadata-self-heal off > 23: option entry-self-heal off > 24: option readdir-failover off > 25: subvolumes hosdu-client-0 hosdu-client-1 > 26: end-volume > 27: > 28: volume hosdu-client-2 > 29: type protocol/client > 30: option remote-host rhsauto019.lab.eng.blr.redhat.com > 31: option remote-subvolume /bricks/hosdu_brick2 > 32: option transport-type socket > 33: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 34: option password d94d42e0-803a-405b-8178-23078d9555b8 > 35: end-volume > 36: > 37: volume hosdu-client-3 > 38: type protocol/client > 39: option remote-host rhsauto008.lab.eng.blr.redhat.com > 40: option remote-subvolume /bricks/hosdu_brick3 > 41: option transport-type socket > 42: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 43: option password d94d42e0-803a-405b-8178-23078d9555b8 > 44: end-volume > 45: > 46: volume hosdu-replicate-1 > 47: type cluster/replicate > 48: option data-self-heal off > 49: option metadata-self-heal off > 50: option entry-self-heal off > 51: option readdir-failover off > 52: subvolumes hosdu-client-2 hosdu-client-3 > 53: end-volume > 54: > 55: volume hosdu-client-4 > 56: type protocol/client > 57: option remote-host rhsauto019.lab.eng.blr.redhat.com > 58: option remote-subvolume /bricks/hosdu_brick4 > 59: option transport-type socket > 60: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 61: option password d94d42e0-803a-405b-8178-23078d9555b8 > 62: end-volume > 63: > 64: volume hosdu-client-5 > 65: type protocol/client > 66: option remote-host rhsauto008.lab.eng.blr.redhat.com > 67: option remote-subvolume /bricks/hosdu_brick5 > 68: option transport-type socket > 69: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 70: option password d94d42e0-803a-405b-8178-23078d9555b8 > 71: end-volume > 72: > 73: volume hosdu-replicate-2 > 74: type cluster/replicate > 75: option data-self-heal off > 76: option metadata-self-heal off > 77: option entry-self-heal off > 78: option readdir-failover off > 79: subvolumes hosdu-client-4 hosdu-client-5 > 80: end-volume > 81: > 82: volume hosdu-client-6 > 83: type protocol/client > 84: option remote-host rhsauto022.lab.eng.blr.redhat.com > 85: option remote-subvolume /bricks/hosdu_brick6 > 86: option transport-type socket > 87: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 88: option password d94d42e0-803a-405b-8178-23078d9555b8 > 89: end-volume > 90: > 91: volume hosdu-client-7 > 92: type protocol/client > 93: option remote-host rhsauto022.lab.eng.blr.redhat.com > 94: option remote-subvolume /bricks/hosdu_brick7 > 95: option transport-type socket > 96: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 97: option password d94d42e0-803a-405b-8178-23078d9555b8 > 98: end-volume > 99: >100: volume hosdu-replicate-3 >101: type cluster/replicate >102: option data-self-heal off >103: option metadata-self-heal off >104: option entry-self-heal off >105: option readdir-failover off >106: subvolumes hosdu-client-6 hosdu-client-7 >107: end-volume >108: >109: volume hosdu-dht >110: type cluster/distribute >111: option use-readdirp yes >112: option lookup-unhashed yes >113: option assert-no-child-down yes >114: option readdir-optimize on >115: option rebalance-cmd 1 >116: option node-uuid 598cd7c0-034b-42a1-9bab-6d84e03fc60f >117: subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3 >118: end-volume >119: >120: volume hosdu-write-behind >121: type performance/write-behind >122: subvolumes hosdu-dht >123: end-volume >124: >125: volume hosdu-read-ahead >126: type performance/read-ahead >127: subvolumes hosdu-write-behind >128: end-volume >129: >130: volume hosdu-io-cache >131: type performance/io-cache >132: subvolumes hosdu-read-ahead >133: end-volume >134: >135: volume hosdu-quick-read >136: type performance/quick-read >137: subvolumes hosdu-io-cache >138: end-volume >139: >140: volume hosdu-md-cache >141: type performance/md-cache >142: subvolumes hosdu-quick-read >143: end-volume >144: >145: volume hosdu >146: type debug/io-stats >147: option latency-measurement on >148: option count-fop-hits on >149: subvolumes hosdu-md-cache >150: end-volume >151: >+------------------------------------------------------------------------------+ >[2013-12-25 07:13:16.029893] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0) >[2013-12-25 07:13:16.034789] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0) >[2013-12-25 07:13:16.034868] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0) >[2013-12-25 07:13:16.042855] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:16.043367] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:16.043459] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:16.043567] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:16.043586] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'. >[2013-12-25 07:13:16.043597] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:16.043804] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online. >[2013-12-25 07:13:16.043856] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:16.043870] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'. >[2013-12-25 07:13:16.043880] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:16.043961] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online. >[2013-12-25 07:13:16.043990] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1 >[2013-12-25 07:13:16.044024] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:16.044037] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49152, attached to remote volume '/bricks/hosdu_brick0'. >[2013-12-25 07:13:16.044046] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:16.044151] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online. >[2013-12-25 07:13:16.044181] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1 >[2013-12-25 07:13:16.044380] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1 >[2013-12-25 07:13:20.821842] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0) >[2013-12-25 07:13:20.829122] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0) >[2013-12-25 07:13:20.836755] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:20.837755] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:20.837793] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'. >[2013-12-25 07:13:20.837810] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:20.837913] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online. >[2013-12-25 07:13:20.838042] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:20.841918] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:20.841966] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49153, attached to remote volume '/bricks/hosdu_brick7'. >[2013-12-25 07:13:20.841986] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:20.842403] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1 >[2013-12-25 07:13:20.843904] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1 >[2013-12-25 07:13:20.844591] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0) >[2013-12-25 07:13:20.848884] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0) >[2013-12-25 07:13:20.853133] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0) >[2013-12-25 07:13:20.858293] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:20.858473] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:20.859152] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:13:20.859231] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:20.859248] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'. >[2013-12-25 07:13:20.859258] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:20.859637] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:20.859658] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'. >[2013-12-25 07:13:20.859668] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:20.859863] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1 >[2013-12-25 07:13:20.860249] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1 >[2013-12-25 07:13:20.861284] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:13:20.861307] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49152, attached to remote volume '/bricks/hosdu_brick1'. >[2013-12-25 07:13:20.861317] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:13:20.862170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1 >[2013-12-25 07:13:20.899196] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode >[2013-12-25 07:13:20.901199] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode >[2013-12-25 07:13:20.901436] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0 >[2013-12-25 07:13:20.901488] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode >[2013-12-25 07:13:20.901841] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-4 >[2013-12-25 07:13:20.901871] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2 >[2013-12-25 07:13:20.902094] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode >[2013-12-25 07:13:20.902667] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of / >[2013-12-25 07:13:20.909876] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on / >[2013-12-25 07:13:20.924881] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:20.925983] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:20.926035] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:20.926089] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:21.338591] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1.00 secs >[2013-12-25 07:13:21.338627] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0 >[2013-12-25 07:13:21.421266] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /4 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:21.443762] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:21.445289] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:21.445422] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:21.445541] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:21.873678] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /29 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:21.879768] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:21.881088] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:21.881215] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:21.881298] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:22.305751] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /30 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:22.333003] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:22.334481] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:22.334571] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:22.334646] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:22.700354] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /74 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:22.707307] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:22.708505] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:22.708612] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:22.708694] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.047313] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /75 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:23.055363] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:23.057321] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.057468] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:23.057553] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.466078] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /76 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:23.483183] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:23.484735] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.484824] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:23.484897] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.875958] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /90 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:13:23.905967] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:23.907601] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.907698] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:23.907771] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.924797] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11) >[2013-12-25 07:13:23.936238] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:23.938008] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.938096] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:23.938169] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.952458] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20) >[2013-12-25 07:13:23.968236] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:23.969958] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.970045] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:23.970117] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:23.989480] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35) >[2013-12-25 07:13:23.998292] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:23.999911] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:23.999998] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.000070] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.015357] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46) >[2013-12-25 07:13:24.028538] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.030451] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.030540] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.030613] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.045415] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49) >[2013-12-25 07:13:24.071092] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.072758] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.072882] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.072995] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.087851] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68) >[2013-12-25 07:13:24.100913] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.102422] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.102511] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.102583] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.115221] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79) >[2013-12-25 07:13:24.122892] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.124368] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.124457] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.124532] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.143799] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85) >[2013-12-25 07:13:24.152105] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.153478] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.153563] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.153636] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.171520] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88) >[2013-12-25 07:13:24.181333] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:13:24.182974] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.183061] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.183133] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.198164] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93) >[2013-12-25 07:13:24.220534] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.222068] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.222155] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.222228] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.241353] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/7) >[2013-12-25 07:13:24.251612] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.253051] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.253138] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.253210] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.269400] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/18) >[2013-12-25 07:13:24.277811] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.279614] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.279701] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.279773] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.296362] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/19) >[2013-12-25 07:13:24.307436] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.308965] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.309053] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.309126] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.325901] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/23) >[2013-12-25 07:13:24.334823] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.336348] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.336439] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.336511] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.354271] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/27) >[2013-12-25 07:13:24.362599] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.364046] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.364135] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.364212] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.379383] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/28) >[2013-12-25 07:13:24.387660] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.388772] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.388821] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.388861] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.404966] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/32) >[2013-12-25 07:13:24.416640] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.417726] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.417815] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.417889] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.434463] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/38) >[2013-12-25 07:13:24.446574] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.447770] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.447859] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.447933] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.466619] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/44) >[2013-12-25 07:13:24.481884] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.483516] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.483605] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.483677] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.501875] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/63) >[2013-12-25 07:13:24.510895] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.512365] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.512455] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.512529] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.527399] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/69) >[2013-12-25 07:13:24.537257] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.538512] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.538599] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.538671] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.553909] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/78) >[2013-12-25 07:13:24.564924] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.566290] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.566392] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.566469] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.582802] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/82) >[2013-12-25 07:13:24.593986] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.595596] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.595682] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.595754] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.612377] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/86) >[2013-12-25 07:13:24.625388] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2013-12-25 07:13:24.626747] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:13:24.626834] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:13:24.626907] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:13:24.641653] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/99) >[2013-12-25 07:13:24.662659] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 3.75 secs >[2013-12-25 07:13:24.682371] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 4.00 secs >[2013-12-25 07:13:24.682417] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 91750400, lookups: 107, failures: 0, skipped: 25 >[2013-12-25 07:13:24.682956] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f621dbe194d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f621e22d851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down >[2013-12-25 07:19:32.917705] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=598cd7c0-034b-42a1-9bab-6d84e03fc60f --socket-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.pid -l /var/log/glusterfs/hosdu-rebalance.log) >[2013-12-25 07:19:32.919594] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 >[2013-12-25 07:19:32.923789] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled >[2013-12-25 07:19:32.923831] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread >[2013-12-25 07:19:32.924100] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled >[2013-12-25 07:19:32.924130] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread >[2013-12-25 07:19:37.926626] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '598cd7c0-034b-42a1-9bab-6d84e03fc60f' >[2013-12-25 07:19:37.926675] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1' >[2013-12-25 07:19:37.926697] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on' >[2013-12-25 07:19:37.926717] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:19:37.926736] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:19:37.926755] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes' >[2013-12-25 07:19:37.926774] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:19:37.926807] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:19:37.926826] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:19:37.926846] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2013-12-25 07:19:37.926866] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:19:37.926885] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:19:37.926903] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:19:37.926922] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2013-12-25 07:19:37.926941] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:19:37.926959] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:19:37.926978] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:19:37.926996] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2013-12-25 07:19:37.927027] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:19:37.927047] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:19:37.927065] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:19:37.927097] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2013-12-25 07:19:37.928184] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2013-12-25 07:19:37.936385] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled >[2013-12-25 07:19:37.936432] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread >[2013-12-25 07:19:37.937080] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled >[2013-12-25 07:19:37.937099] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread >[2013-12-25 07:19:37.937746] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled >[2013-12-25 07:19:37.937764] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread >[2013-12-25 07:19:37.938531] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled >[2013-12-25 07:19:37.938552] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread >[2013-12-25 07:19:37.939292] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled >[2013-12-25 07:19:37.939312] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread >[2013-12-25 07:19:37.939950] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled >[2013-12-25 07:19:37.939967] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread >[2013-12-25 07:19:37.940638] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled >[2013-12-25 07:19:37.940655] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread >[2013-12-25 07:19:37.941306] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled >[2013-12-25 07:19:37.941324] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread >[2013-12-25 07:19:37.941394] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.945807] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.949923] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.953857] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.958019] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.962275] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.966540] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport >[2013-12-25 07:19:37.970952] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume hosdu-client-0 > 2: type protocol/client > 3: option remote-host rhsauto019.lab.eng.blr.redhat.com > 4: option remote-subvolume /bricks/hosdu_brick2 > 5: option transport-type socket > 6: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 7: option password d94d42e0-803a-405b-8178-23078d9555b8 > 8: end-volume > 9: > 10: volume hosdu-client-1 > 11: type protocol/client > 12: option remote-host rhsauto008.lab.eng.blr.redhat.com > 13: option remote-subvolume /bricks/hosdu_brick3 > 14: option transport-type socket > 15: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 16: option password d94d42e0-803a-405b-8178-23078d9555b8 > 17: end-volume > 18: > 19: volume hosdu-replicate-0 > 20: type cluster/replicate > 21: option data-self-heal off > 22: option metadata-self-heal off > 23: option entry-self-heal off > 24: option readdir-failover off > 25: subvolumes hosdu-client-0 hosdu-client-1 > 26: end-volume > 27: > 28: volume hosdu-client-2 > 29: type protocol/client > 30: option remote-host rhsauto019.lab.eng.blr.redhat.com > 31: option remote-subvolume /bricks/hosdu_brick4 > 32: option transport-type socket > 33: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 34: option password d94d42e0-803a-405b-8178-23078d9555b8 > 35: end-volume > 36: > 37: volume hosdu-client-3 > 38: type protocol/client > 39: option remote-host rhsauto008.lab.eng.blr.redhat.com > 40: option remote-subvolume /bricks/hosdu_brick5 > 41: option transport-type socket > 42: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 43: option password d94d42e0-803a-405b-8178-23078d9555b8 > 44: end-volume > 45: > 46: volume hosdu-replicate-1 > 47: type cluster/replicate > 48: option data-self-heal off > 49: option metadata-self-heal off > 50: option entry-self-heal off > 51: option readdir-failover off > 52: subvolumes hosdu-client-2 hosdu-client-3 > 53: end-volume > 54: > 55: volume hosdu-client-4 > 56: type protocol/client > 57: option remote-host rhsauto022.lab.eng.blr.redhat.com > 58: option remote-subvolume /bricks/hosdu_brick6 > 59: option transport-type socket > 60: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 61: option password d94d42e0-803a-405b-8178-23078d9555b8 > 62: end-volume > 63: > 64: volume hosdu-client-5 > 65: type protocol/client > 66: option remote-host rhsauto022.lab.eng.blr.redhat.com > 67: option remote-subvolume /bricks/hosdu_brick8 > 68: option transport-type socket > 69: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 70: option password d94d42e0-803a-405b-8178-23078d9555b8 > 71: end-volume > 72: > 73: volume hosdu-replicate-2 > 74: type cluster/replicate > 75: option data-self-heal off > 76: option metadata-self-heal off > 77: option entry-self-heal off > 78: option readdir-failover off > 79: subvolumes hosdu-client-4 hosdu-client-5 > 80: end-volume > 81: > 82: volume hosdu-client-6 > 83: type protocol/client > 84: option remote-host rhsauto022.lab.eng.blr.redhat.com > 85: option remote-subvolume /bricks/hosdu_brick9 > 86: option transport-type socket > 87: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 88: option password d94d42e0-803a-405b-8178-23078d9555b8 > 89: end-volume > 90: > 91: volume hosdu-client-7 > 92: type protocol/client > 93: option remote-host rhsauto022.lab.eng.blr.redhat.com > 94: option remote-subvolume /bricks/hosdu_brick10 > 95: option transport-type socket > 96: option username 4c71e3df-528c-460b-a81c-0da8a731de82 > 97: option password d94d42e0-803a-405b-8178-23078d9555b8 > 98: end-volume > 99: >100: volume hosdu-replicate-3 >101: type cluster/replicate >102: option data-self-heal off >103: option metadata-self-heal off >104: option entry-self-heal off >105: option readdir-failover off >106: subvolumes hosdu-client-6 hosdu-client-7 >107: end-volume >108: >109: volume hosdu-dht >110: type cluster/distribute >111: option use-readdirp yes >112: option lookup-unhashed yes >113: option assert-no-child-down yes >114: option readdir-optimize on >115: option rebalance-cmd 1 >116: option node-uuid 598cd7c0-034b-42a1-9bab-6d84e03fc60f >117: subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3 >118: end-volume >119: >120: volume hosdu-write-behind >121: type performance/write-behind >122: subvolumes hosdu-dht >123: end-volume >124: >125: volume hosdu-read-ahead >126: type performance/read-ahead >127: subvolumes hosdu-write-behind >128: end-volume >129: >130: volume hosdu-io-cache >131: type performance/io-cache >132: subvolumes hosdu-read-ahead >133: end-volume >134: >135: volume hosdu-quick-read >136: type performance/quick-read >137: subvolumes hosdu-io-cache >138: end-volume >139: >140: volume hosdu-md-cache >141: type performance/md-cache >142: subvolumes hosdu-quick-read >143: end-volume >144: >145: volume hosdu >146: type debug/io-stats >147: option latency-measurement on >148: option count-fop-hits on >149: subvolumes hosdu-md-cache >150: end-volume >151: >+------------------------------------------------------------------------------+ >[2013-12-25 07:19:37.977203] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49153 (from 0) >[2013-12-25 07:19:37.982144] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49154 (from 0) >[2013-12-25 07:19:37.986806] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:37.987319] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:37.987353] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'. >[2013-12-25 07:19:37.987365] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:37.987574] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online. >[2013-12-25 07:19:37.987618] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:37.987849] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1 >[2013-12-25 07:19:37.987883] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:37.987896] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'. >[2013-12-25 07:19:37.987906] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:37.988127] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online. >[2013-12-25 07:19:37.988174] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1 >[2013-12-25 07:19:42.941511] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49152 (from 0) >[2013-12-25 07:19:42.948824] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49155 (from 0) >[2013-12-25 07:19:42.948942] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49157 (from 0) >[2013-12-25 07:19:42.962322] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49156 (from 0) >[2013-12-25 07:19:42.969719] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49153 (from 0) >[2013-12-25 07:19:42.969833] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49154 (from 0) >[2013-12-25 07:19:42.979500] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.979648] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.979728] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.980157] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.980259] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.980324] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2013-12-25 07:19:42.980574] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.980599] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'. >[2013-12-25 07:19:42.980612] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.980715] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online. >[2013-12-25 07:19:42.980760] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.980774] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'. >[2013-12-25 07:19:42.980783] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.981300] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1 >[2013-12-25 07:19:42.981338] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.981357] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.252:49155, attached to remote volume '/bricks/hosdu_brick8'. >[2013-12-25 07:19:42.981375] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.981571] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.981597] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49157, attached to remote volume '/bricks/hosdu_brick10'. >[2013-12-25 07:19:42.981607] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.981661] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-7' came back up; going online. >[2013-12-25 07:19:42.981776] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.981794] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'. >[2013-12-25 07:19:42.981804] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.982195] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1 >[2013-12-25 07:19:42.982225] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1 >[2013-12-25 07:19:42.982664] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1 >[2013-12-25 07:19:42.982893] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client >[2013-12-25 07:19:42.982912] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49156, attached to remote volume '/bricks/hosdu_brick9'. >[2013-12-25 07:19:42.982931] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2013-12-25 07:19:42.983221] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1 >[2013-12-25 07:19:42.984296] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1 >[2013-12-25 07:19:42.987941] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode >[2013-12-25 07:19:42.989762] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode >[2013-12-25 07:19:42.990166] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0 >[2013-12-25 07:19:42.990213] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2 >[2013-12-25 07:19:42.990330] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode >[2013-12-25 07:19:42.990475] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode >[2013-12-25 07:19:42.991056] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of / >[2013-12-25 07:19:42.995686] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on / >[2013-12-25 07:19:43.011072] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.012152] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.012224] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.012270] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.025113] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/4) >[2013-12-25 07:19:43.047574] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.048654] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.048743] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.048815] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.063133] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/29) >[2013-12-25 07:19:43.074996] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.075748] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.075801] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.075847] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.087315] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/30) >[2013-12-25 07:19:43.111618] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.112845] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.112896] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.112940] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.126487] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/74) >[2013-12-25 07:19:43.134756] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.135365] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.135424] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.135469] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.155706] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/75) >[2013-12-25 07:19:43.163080] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.164333] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.164411] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.164453] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.177073] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/76) >[2013-12-25 07:19:43.189157] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2013-12-25 07:19:43.190252] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.190308] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.190351] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.218089] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/90) >[2013-12-25 07:19:43.232754] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.233811] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.233860] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.233904] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.258905] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11) >[2013-12-25 07:19:43.268690] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.269646] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.269730] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.269784] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.305107] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20) >[2013-12-25 07:19:43.318424] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.319459] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.319512] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.319556] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.341160] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35) >[2013-12-25 07:19:43.350991] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.351989] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.352037] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.352100] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.371875] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1.00 secs >[2013-12-25 07:19:43.371939] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 47, failures: 0, skipped: 10 >[2013-12-25 07:19:43.386498] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46) >[2013-12-25 07:19:43.407020] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.408409] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.408506] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.408587] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.433863] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49) >[2013-12-25 07:19:43.463299] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.464748] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.464867] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.464982] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.492830] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68) >[2013-12-25 07:19:43.507784] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.509392] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.509519] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.509605] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.547276] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79) >[2013-12-25 07:19:43.560698] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.561925] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.562013] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.562085] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.590444] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85) >[2013-12-25 07:19:43.599932] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.601471] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.601578] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.601653] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.623834] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88) >[2013-12-25 07:19:43.635665] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2013-12-25 07:19:43.636906] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal >[2013-12-25 07:19:43.636993] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal >[2013-12-25 07:19:43.637069] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal >[2013-12-25 07:19:43.666274] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93) >[2013-12-25 07:19:43.793311] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.80 secs >[2013-12-25 07:19:43.822636] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs >[2013-12-25 07:19:43.822761] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 17 >[2013-12-25 07:19:43.823349] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f9d3e99794d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f9d3efe3851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1046603
: 841817 |
841881
|
841882