Attachment 841817 Details for Bug 1046603 – Rebalance logs from masternode

Rebalance logs from masternode

hosdu-rebalance.log (text/x-log), 114.55 KB, created by Lalatendu Mohanty on 2013-12-26 09:51:40 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2013-12-26 09:51:40 UTC

Size: 114.55 KB

patch

obsolete

>[2013-12-25 07:13:10.806923] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=598cd7c0-034b-42a1-9bab-6d84e03fc60f --socket-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 07:13:10.808637] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 07:13:10.812636] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 07:13:10.812678] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 07:13:10.812950] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 07:13:10.812981] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 07:13:15.812715] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '598cd7c0-034b-42a1-9bab-6d84e03fc60f'
>[2013-12-25 07:13:15.812758] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 07:13:15.812778] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 07:13:15.812797] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:15.812815] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:15.812832] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:15.812850] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:15.812880] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:15.812898] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:15.812915] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:15.812933] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:15.812950] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:15.812967] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:15.812984] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:15.813015] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:15.813046] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:15.813063] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:15.813091] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:15.813111] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:15.813128] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:15.813145] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:15.813193] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:15.814037] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 07:13:15.858679] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 07:13:15.858732] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 07:13:15.862585] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 07:13:15.862606] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 07:13:15.866164] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 07:13:15.866205] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 07:13:15.870271] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 07:13:15.870291] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 07:13:15.874804] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 07:13:15.874826] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 07:13:15.878846] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 07:13:15.878868] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 07:13:15.882507] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 07:13:15.882529] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 07:13:15.886010] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 07:13:15.886031] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 07:13:15.886106] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:15.893695] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:15.989654] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:15.996572] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:16.003467] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:16.010297] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:16.017053] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:16.023995] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick0
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick1
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick2
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick3
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick4
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick5
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick6
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick7
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 598cd7c0-034b-42a1-9bab-6d84e03fc60f
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 07:13:16.029893] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2013-12-25 07:13:16.034789] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0)
>[2013-12-25 07:13:16.034868] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2013-12-25 07:13:16.042855] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:16.043367] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:16.043459] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:16.043567] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:16.043586] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 07:13:16.043597] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:16.043804] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2013-12-25 07:13:16.043856] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:16.043870] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 07:13:16.043880] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:16.043961] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online.
>[2013-12-25 07:13:16.043990] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 07:13:16.044024] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:16.044037] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49152, attached to remote volume '/bricks/hosdu_brick0'.
>[2013-12-25 07:13:16.044046] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:16.044151] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-25 07:13:16.044181] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 07:13:16.044380] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 07:13:20.821842] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2013-12-25 07:13:20.829122] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0)
>[2013-12-25 07:13:20.836755] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:20.837755] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:20.837793] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 07:13:20.837810] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:20.837913] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-25 07:13:20.838042] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:20.841918] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:20.841966] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49153, attached to remote volume '/bricks/hosdu_brick7'.
>[2013-12-25 07:13:20.841986] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:20.842403] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 07:13:20.843904] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 07:13:20.844591] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0)
>[2013-12-25 07:13:20.848884] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2013-12-25 07:13:20.853133] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2013-12-25 07:13:20.858293] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:20.858473] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:20.859152] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:20.859231] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:20.859248] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 07:13:20.859258] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:20.859637] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:20.859658] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 07:13:20.859668] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:20.859863] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 07:13:20.860249] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 07:13:20.861284] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:20.861307] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49152, attached to remote volume '/bricks/hosdu_brick1'.
>[2013-12-25 07:13:20.861317] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:20.862170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 07:13:20.899196] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 07:13:20.901199] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 07:13:20.901436] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0
>[2013-12-25 07:13:20.901488] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 07:13:20.901841] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-4
>[2013-12-25 07:13:20.901871] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2
>[2013-12-25 07:13:20.902094] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 07:13:20.902667] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 07:13:20.909876] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 07:13:20.924881] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:20.925983] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:20.926035] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:20.926089] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:21.338591] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1.00 secs
>[2013-12-25 07:13:21.338627] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2013-12-25 07:13:21.421266] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /4 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:21.443762] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:21.445289] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:21.445422] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:21.445541] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:21.873678] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /29 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:21.879768] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:21.881088] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:21.881215] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:21.881298] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:22.305751] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /30 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:22.333003] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:22.334481] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:22.334571] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:22.334646] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:22.700354] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /74 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:22.707307] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:22.708505] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:22.708612] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:22.708694] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.047313] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /75 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:23.055363] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:23.057321] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.057468] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:23.057553] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.466078] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /76 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:23.483183] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:23.484735] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.484824] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:23.484897] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.875958] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /90 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:13:23.905967] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:23.907601] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.907698] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:23.907771] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.924797] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11)
>[2013-12-25 07:13:23.936238] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:23.938008] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.938096] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:23.938169] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.952458] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20)
>[2013-12-25 07:13:23.968236] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:23.969958] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.970045] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:23.970117] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:23.989480] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35)
>[2013-12-25 07:13:23.998292] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:23.999911] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:23.999998] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.000070] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.015357] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46)
>[2013-12-25 07:13:24.028538] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.030451] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.030540] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.030613] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.045415] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49)
>[2013-12-25 07:13:24.071092] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.072758] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.072882] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.072995] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.087851] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68)
>[2013-12-25 07:13:24.100913] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.102422] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.102511] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.102583] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.115221] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79)
>[2013-12-25 07:13:24.122892] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.124368] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.124457] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.124532] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.143799] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85)
>[2013-12-25 07:13:24.152105] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.153478] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.153563] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.153636] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.171520] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88)
>[2013-12-25 07:13:24.181333] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:13:24.182974] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.183061] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.183133] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.198164] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93)
>[2013-12-25 07:13:24.220534] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.222068] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.222155] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.222228] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.241353] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/7)
>[2013-12-25 07:13:24.251612] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.253051] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.253138] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.253210] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.269400] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/18)
>[2013-12-25 07:13:24.277811] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.279614] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.279701] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.279773] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.296362] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/19)
>[2013-12-25 07:13:24.307436] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.308965] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.309053] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.309126] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.325901] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/23)
>[2013-12-25 07:13:24.334823] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.336348] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.336439] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.336511] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.354271] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/27)
>[2013-12-25 07:13:24.362599] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.364046] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.364135] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.364212] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.379383] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/28)
>[2013-12-25 07:13:24.387660] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.388772] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.388821] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.388861] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.404966] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/32)
>[2013-12-25 07:13:24.416640] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.417726] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.417815] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.417889] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.434463] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/38)
>[2013-12-25 07:13:24.446574] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.447770] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.447859] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.447933] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.466619] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/44)
>[2013-12-25 07:13:24.481884] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.483516] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.483605] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.483677] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.501875] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/63)
>[2013-12-25 07:13:24.510895] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.512365] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.512455] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.512529] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.527399] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/69)
>[2013-12-25 07:13:24.537257] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.538512] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.538599] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.538671] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.553909] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/78)
>[2013-12-25 07:13:24.564924] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.566290] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.566392] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.566469] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.582802] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/82)
>[2013-12-25 07:13:24.593986] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.595596] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.595682] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.595754] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.612377] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/86)
>[2013-12-25 07:13:24.625388] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 07:13:24.626747] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f6213ba8e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:13:24.626834] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f6213ba8e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:13:24.626907] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f6213913a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f621eac6c99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f6213ba8e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:13:24.641653] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/99)
>[2013-12-25 07:13:24.662659] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 3.75 secs
>[2013-12-25 07:13:24.682371] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 4.00 secs
>[2013-12-25 07:13:24.682417] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 91750400, lookups: 107, failures: 0, skipped: 25
>[2013-12-25 07:13:24.682956] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f621dbe194d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f621e22d851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down
>[2013-12-25 07:19:32.917705] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=598cd7c0-034b-42a1-9bab-6d84e03fc60f --socket-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/598cd7c0-034b-42a1-9bab-6d84e03fc60f.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 07:19:32.919594] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 07:19:32.923789] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 07:19:32.923831] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 07:19:32.924100] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 07:19:32.924130] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 07:19:37.926626] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '598cd7c0-034b-42a1-9bab-6d84e03fc60f'
>[2013-12-25 07:19:37.926675] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 07:19:37.926697] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 07:19:37.926717] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:37.926736] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:37.926755] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:37.926774] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:37.926807] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:37.926826] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:37.926846] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:37.926866] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:37.926885] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:37.926903] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:37.926922] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:37.926941] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:37.926959] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:37.926978] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:37.926996] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:37.927027] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:37.927047] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:37.927065] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:37.927097] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:37.928184] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 07:19:37.936385] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 07:19:37.936432] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 07:19:37.937080] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 07:19:37.937099] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 07:19:37.937746] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 07:19:37.937764] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 07:19:37.938531] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 07:19:37.938552] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 07:19:37.939292] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 07:19:37.939312] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 07:19:37.939950] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 07:19:37.939967] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 07:19:37.940638] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 07:19:37.940655] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 07:19:37.941306] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 07:19:37.941324] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 07:19:37.941394] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.945807] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.949923] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.953857] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.958019] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.962275] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.966540] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:37.970952] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick2
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick3
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick4
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick5
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick6
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick8
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick9
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick10
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 598cd7c0-034b-42a1-9bab-6d84e03fc60f
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 07:19:37.977203] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49153 (from 0)
>[2013-12-25 07:19:37.982144] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49154 (from 0)
>[2013-12-25 07:19:37.986806] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:37.987319] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:37.987353] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 07:19:37.987365] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:37.987574] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-25 07:19:37.987618] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:37.987849] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 07:19:37.987883] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:37.987896] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 07:19:37.987906] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:37.988127] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2013-12-25 07:19:37.988174] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 07:19:42.941511] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49152 (from 0)
>[2013-12-25 07:19:42.948824] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49155 (from 0)
>[2013-12-25 07:19:42.948942] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49157 (from 0)
>[2013-12-25 07:19:42.962322] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49156 (from 0)
>[2013-12-25 07:19:42.969719] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49153 (from 0)
>[2013-12-25 07:19:42.969833] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49154 (from 0)
>[2013-12-25 07:19:42.979500] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.979648] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.979728] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.980157] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.980259] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.980324] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:42.980574] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.980599] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 07:19:42.980612] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.980715] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online.
>[2013-12-25 07:19:42.980760] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.980774] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 07:19:42.980783] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.981300] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 07:19:42.981338] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.981357] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.252:49155, attached to remote volume '/bricks/hosdu_brick8'.
>[2013-12-25 07:19:42.981375] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.981571] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.981597] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49157, attached to remote volume '/bricks/hosdu_brick10'.
>[2013-12-25 07:19:42.981607] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.981661] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-7' came back up; going online.
>[2013-12-25 07:19:42.981776] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.981794] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 07:19:42.981804] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.982195] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 07:19:42.982225] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 07:19:42.982664] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 07:19:42.982893] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:42.982912] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49156, attached to remote volume '/bricks/hosdu_brick9'.
>[2013-12-25 07:19:42.982931] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:42.983221] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 07:19:42.984296] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 07:19:42.987941] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 07:19:42.989762] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 07:19:42.990166] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0
>[2013-12-25 07:19:42.990213] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2
>[2013-12-25 07:19:42.990330] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 07:19:42.990475] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 07:19:42.991056] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 07:19:42.995686] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 07:19:43.011072] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.012152] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.012224] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.012270] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.025113] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/4)
>[2013-12-25 07:19:43.047574] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.048654] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.048743] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.048815] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.063133] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/29)
>[2013-12-25 07:19:43.074996] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.075748] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.075801] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.075847] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.087315] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/30)
>[2013-12-25 07:19:43.111618] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.112845] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.112896] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.112940] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.126487] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/74)
>[2013-12-25 07:19:43.134756] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.135365] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.135424] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.135469] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.155706] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/75)
>[2013-12-25 07:19:43.163080] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.164333] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.164411] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.164453] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.177073] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/76)
>[2013-12-25 07:19:43.189157] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-25 07:19:43.190252] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.190308] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.190351] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.218089] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/90)
>[2013-12-25 07:19:43.232754] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.233811] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.233860] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.233904] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.258905] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11)
>[2013-12-25 07:19:43.268690] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.269646] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.269730] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.269784] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.305107] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20)
>[2013-12-25 07:19:43.318424] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.319459] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.319512] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.319556] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.341160] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35)
>[2013-12-25 07:19:43.350991] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.351989] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.352037] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.352100] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.371875] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1.00 secs
>[2013-12-25 07:19:43.371939] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 47, failures: 0, skipped: 10
>[2013-12-25 07:19:43.386498] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46)
>[2013-12-25 07:19:43.407020] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.408409] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.408506] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.408587] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.433863] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49)
>[2013-12-25 07:19:43.463299] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.464748] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.464867] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.464982] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.492830] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68)
>[2013-12-25 07:19:43.507784] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.509392] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.509519] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.509605] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.547276] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79)
>[2013-12-25 07:19:43.560698] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.561925] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.562013] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.562085] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.590444] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85)
>[2013-12-25 07:19:43.599932] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.601471] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.601578] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.601653] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.623834] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88)
>[2013-12-25 07:19:43.635665] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-25 07:19:43.636906] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f9d38a7ae54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 07:19:43.636993] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f9d38a7ae79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 07:19:43.637069] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f9d387e5a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x7f9d3f87cc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f9d38a7ae9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 07:19:43.666274] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93)
>[2013-12-25 07:19:43.793311] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.80 secs
>[2013-12-25 07:19:43.822636] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2013-12-25 07:19:43.822761] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 17
>[2013-12-25 07:19:43.823349] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f9d3e99794d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f9d3efe3851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1046603: 841817 | 841881 | 841882