Attachment 832062 Details for Bug 1037515 – Rebalance Log

Rebalance Log

hosdu-rebalance.log (text/x-log), 71.24 KB, created by Lalatendu Mohanty on 2013-12-03 13:41:14 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2013-12-03 13:41:14 UTC

Size: 71.24 KB

patch

obsolete

>[2013-12-03 03:09:57.256380] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=1c805ca6-45cc-444a-aa8a-ebffd4192157 --socket-file /var/lib/glusterd/vols/hosdu/rebalance/1c805ca6-45cc-444a-aa8a-ebffd4192157.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/1c805ca6-45cc-444a-aa8a-ebffd4192157.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-03 03:09:57.258108] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-03 03:09:57.261515] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-03 03:09:57.261546] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-03 03:09:57.261817] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-03 03:09:57.261848] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-03 03:10:02.257964] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '1c805ca6-45cc-444a-aa8a-ebffd4192157'
>[2013-12-03 03:10:02.258009] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-03 03:10:02.258030] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-03 03:10:02.258050] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:10:02.258068] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:10:02.258086] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:10:02.258104] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:10:02.258135] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:10:02.258154] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:10:02.258173] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:10:02.258191] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:10:02.258209] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:10:02.258227] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:10:02.258244] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:10:02.258262] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:10:02.258279] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:10:02.258296] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:10:02.258325] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:10:02.258345] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:10:02.258363] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:10:02.258381] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:10:02.258410] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:10:02.259153] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-03 03:10:02.282572] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-03 03:10:02.282620] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-03 03:10:02.283752] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-03 03:10:02.283773] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-03 03:10:02.284629] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-03 03:10:02.284650] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-03 03:10:02.285522] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-03 03:10:02.285543] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-03 03:10:02.286479] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-03 03:10:02.286499] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-03 03:10:02.287678] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-03 03:10:02.287699] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-03 03:10:02.289438] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-03 03:10:02.289482] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-03 03:10:02.290995] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-03 03:10:02.291015] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-03 03:10:02.291074] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.298232] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.302488] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.306137] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.309679] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.313255] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.316980] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-03 03:10:02.320696] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick0
>  5:     option transport-type socket
>  6:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
>  7:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick1
> 14:     option transport-type socket
> 15:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 16:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick2
> 32:     option transport-type socket
> 33:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 34:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick3
> 41:     option transport-type socket
> 42:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 43:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick4
> 59:     option transport-type socket
> 60:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 61:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick5
> 68:     option transport-type socket
> 69:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 70:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick6
> 86:     option transport-type socket
> 87:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 88:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick7
> 95:     option transport-type socket
> 96:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 97:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 1c805ca6-45cc-444a-aa8a-ebffd4192157
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-03 03:10:02.326178] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0)
>[2013-12-03 03:10:02.326292] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2013-12-03 03:10:02.326379] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2013-12-03 03:10:02.338029] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:02.338315] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:02.338609] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:02.338639] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49152, attached to remote volume '/bricks/hosdu_brick0'.
>[2013-12-03 03:10:02.338650] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:02.338863] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-03 03:10:02.338919] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:02.338933] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-03 03:10:02.338942] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:02.339143] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2013-12-03 03:10:02.339179] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:02.339264] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-03 03:10:02.339290] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-03 03:10:02.339865] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:02.339886] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-03 03:10:02.339896] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:02.339980] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online.
>[2013-12-03 03:10:02.340275] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-03 03:10:07.266851] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2013-12-03 03:10:07.271966] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0)
>[2013-12-03 03:10:07.282765] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:07.282987] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0)
>[2013-12-03 03:10:07.283074] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2013-12-03 03:10:07.293675] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2013-12-03 03:10:07.293736] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:07.297670] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:07.297698] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-03 03:10:07.297709] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:07.297774] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-03 03:10:07.297925] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:07.297943] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49153, attached to remote volume '/bricks/hosdu_brick7'.
>[2013-12-03 03:10:07.297975] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:07.298282] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:07.298410] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-03 03:10:07.298481] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-03 03:10:07.298647] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:07.298728] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:10:07.299684] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:07.299711] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49152, attached to remote volume '/bricks/hosdu_brick1'.
>[2013-12-03 03:10:07.299723] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:07.300043] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:07.300081] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-03 03:10:07.300093] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:07.300256] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:10:07.300274] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-03 03:10:07.300294] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:10:07.300852] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-03 03:10:07.300902] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-03 03:10:07.301830] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-03 03:10:07.322690] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-03 03:10:07.324244] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-03 03:10:07.324431] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-03 03:10:07.324593] I [afr-common.c:2274:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2
>[2013-12-03 03:10:07.324642] I [afr-common.c:2274:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0
>[2013-12-03 03:10:07.324864] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-03 03:10:07.325068] I [afr-common.c:2274:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-4
>[2013-12-03 03:10:07.325693] I [dht-common.c:2645:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-03 03:10:07.330000] I [dht-rebalance.c:1117:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-03 03:10:07.346910] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:07.727765] I [dht-rebalance.c:1776:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2013-12-03 03:10:07.727874] I [dht-rebalance.c:1779:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2013-12-03 03:10:07.955893] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /4 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:07.958251] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /6: failed to get trusted.distribute.linkinfo key - No such file or directory
>[2013-12-03 03:10:07.960123] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /10: failed to get trusted.distribute.linkinfo key - No such file or directory
>[2013-12-03 03:10:07.965003] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /14: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:07.966809] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /15: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:07.978047] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:08.931580] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /29 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:08.936669] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:09.762453] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /30 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:09.765195] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /31: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.767508] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /36: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.769881] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /39: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.772052] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /40: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.774223] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /42: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.776552] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /45: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.779066] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /50: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.781275] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /52: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.783464] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /55: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.785640] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /59: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.787649] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /66: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.789701] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /67: failed to get trusted.distribute.linkinfo key - Invalid argument
>[2013-12-03 03:10:09.797329] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:10.763234] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /74 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:10.770635] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:11.475229] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /75 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:11.484029] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:11.974994] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /76 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:11.988097] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:12.542803] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /90 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:10:12.569191] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.581020] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11)
>[2013-12-03 03:10:12.583511] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /13: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.590159] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.604873] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20)
>[2013-12-03 03:10:12.619717] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.634106] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35)
>[2013-12-03 03:10:12.641732] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.653891] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46)
>[2013-12-03 03:10:12.665429] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.677716] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49)
>[2013-12-03 03:10:12.688989] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /61: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.691042] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /62: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.692986] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /64: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.695344] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /65: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.702652] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.713954] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68)
>[2013-12-03 03:10:12.727258] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.739717] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79)
>[2013-12-03 03:10:12.747492] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.761440] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85)
>[2013-12-03 03:10:12.768324] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.780735] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88)
>[2013-12-03 03:10:12.782750] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /91: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.788686] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:10:12.801498] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93)
>[2013-12-03 03:10:12.803207] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /94: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.804548] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /96: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.805999] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /97: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.807425] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /98: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.827297] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.839274] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/7)
>[2013-12-03 03:10:12.847075] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.857507] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/18)
>[2013-12-03 03:10:12.862498] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.874576] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/19)
>[2013-12-03 03:10:12.883348] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.893393] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/23)
>[2013-12-03 03:10:12.898168] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.908797] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/27)
>[2013-12-03 03:10:12.913495] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.923221] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/28)
>[2013-12-03 03:10:12.928144] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.942304] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/32)
>[2013-12-03 03:10:12.953633] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.963912] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/38)
>[2013-12-03 03:10:12.972995] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.981673] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/44)
>[2013-12-03 03:10:12.983165] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /56: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.984233] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /57: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.985281] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /60: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:12.989601] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:12.997955] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/63)
>[2013-12-03 03:10:13.002576] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:13.011969] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/69)
>[2013-12-03 03:10:13.013587] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /71: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:13.018023] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:13.026146] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/78)
>[2013-12-03 03:10:13.027767] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /80: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:10:13.033932] I [dht-rebalance.c:1776:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 6.00 secs
>[2013-12-03 03:10:13.033976] I [dht-rebalance.c:1779:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 91750400, lookups: 95, failures: 31, skipped: 22
>[2013-12-03 03:10:13.034231] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:13.047861] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/82)
>[2013-12-03 03:10:13.056427] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:13.066865] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/86)
>[2013-12-03 03:10:13.079129] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-03 03:10:13.090104] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-2) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/99)
>[2013-12-03 03:10:13.111403] I [dht-rebalance.c:1341:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 5.78 secs
>[2013-12-03 03:10:13.130250] I [dht-rebalance.c:1776:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 6.00 secs
>[2013-12-03 03:10:13.130298] I [dht-rebalance.c:1779:gf_defrag_status_get] 0-glusterfs: Files migrated: 7, size: 91750400, lookups: 107, failures: 31, skipped: 25
>[2013-12-03 03:10:13.131332] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3348ce894d] (-->/lib64/libpthread.so.0() [0x3349407851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down
>[2013-12-03 03:16:12.722616] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=1c805ca6-45cc-444a-aa8a-ebffd4192157 --socket-file /var/lib/glusterd/vols/hosdu/rebalance/1c805ca6-45cc-444a-aa8a-ebffd4192157.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/1c805ca6-45cc-444a-aa8a-ebffd4192157.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-03 03:16:12.724196] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-03 03:16:12.729398] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-03 03:16:12.729457] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-03 03:16:12.729846] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-03 03:16:12.729871] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-03 03:16:17.722151] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '1c805ca6-45cc-444a-aa8a-ebffd4192157'
>[2013-12-03 03:16:17.722183] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-03 03:16:17.722195] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-03 03:16:17.722206] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:16:17.722216] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:16:17.722226] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-03 03:16:17.722236] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:16:17.722253] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:16:17.722264] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:16:17.722274] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-03 03:16:17.722284] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:16:17.722294] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:16:17.722304] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:16:17.722313] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-03 03:16:17.722323] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:16:17.722332] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:16:17.722342] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:16:17.722351] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-03 03:16:17.722361] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:16:17.722370] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:16:17.722380] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:16:17.722396] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-03 03:16:17.722859] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-03 03:16:17.728445] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-03 03:16:17.728493] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-03 03:16:17.729270] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-03 03:16:17.729290] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-03 03:16:17.730009] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-03 03:16:17.730028] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-03 03:16:17.730734] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-03 03:16:17.730752] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-03 03:16:17.731413] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-03 03:16:17.731429] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-03 03:16:17.732281] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-03 03:16:17.732301] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-03 03:16:17.733025] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-03 03:16:17.733044] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-03 03:16:17.733772] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-03 03:16:17.733792] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-03 03:16:17.733854] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:17.740531] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.204052] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.212302] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.216651] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.222072] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.226400] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-03 03:16:18.231393] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick2
>  5:     option transport-type socket
>  6:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
>  7:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick3
> 14:     option transport-type socket
> 15:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 16:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick4
> 32:     option transport-type socket
> 33:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 34:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick5
> 41:     option transport-type socket
> 42:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 43:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick6
> 59:     option transport-type socket
> 60:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 61:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick8
> 68:     option transport-type socket
> 69:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 70:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick9
> 86:     option transport-type socket
> 87:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 88:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick10
> 95:     option transport-type socket
> 96:     option username f38f3f89-d89a-46db-a890-fc90ac9e2178
> 97:     option password b7c7bd39-2481-424a-a36f-97d0618bf07a
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 1c805ca6-45cc-444a-aa8a-ebffd4192157
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-03 03:16:18.239086] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49153 (from 0)
>[2013-12-03 03:16:18.245661] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49154 (from 0)
>[2013-12-03 03:16:18.251500] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:18.251910] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:18.251933] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-03 03:16:18.251944] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:18.252176] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-03 03:16:18.252239] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-03 03:16:18.252267] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:18.252919] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:18.252942] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-03 03:16:18.252953] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:18.253039] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2013-12-03 03:16:18.253329] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-03 03:16:22.745776] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49155 (from 0)
>[2013-12-03 03:16:22.754526] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49152 (from 0)
>[2013-12-03 03:16:22.762381] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49156 (from 0)
>[2013-12-03 03:16:22.762496] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49157 (from 0)
>[2013-12-03 03:16:22.775846] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.776447] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49153 (from 0)
>[2013-12-03 03:16:22.776567] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49154 (from 0)
>[2013-12-03 03:16:22.784030] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.784166] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.784250] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.784335] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.784353] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.252:49155, attached to remote volume '/bricks/hosdu_brick8'.
>[2013-12-03 03:16:22.784363] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.784440] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-5' came back up; going online.
>[2013-12-03 03:16:22.784816] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.785060] I [client-handshake.c:1676:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-03 03:16:22.785174] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.785192] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49156, attached to remote volume '/bricks/hosdu_brick9'.
>[2013-12-03 03:16:22.785202] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.785255] I [afr-common.c:3885:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-03 03:16:22.785415] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.785432] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-03 03:16:22.785441] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.785917] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-03 03:16:22.785973] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.785996] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-03 03:16:22.786006] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.786286] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.786341] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49157, attached to remote volume '/bricks/hosdu_brick10'.
>[2013-12-03 03:16:22.786390] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.786621] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-03 03:16:22.786671] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-03 03:16:22.786684] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-03 03:16:22.786694] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-03 03:16:22.786906] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-03 03:16:22.787303] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-03 03:16:22.788022] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-03 03:16:22.788532] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-03 03:16:22.791740] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-03 03:16:22.793715] I [afr-common.c:2274:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-0
>[2013-12-03 03:16:22.793805] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-03 03:16:22.794151] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-03 03:16:22.794285] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-03 03:16:22.794530] I [afr-common.c:2274:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-2
>[2013-12-03 03:16:22.794852] I [dht-common.c:2645:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-03 03:16:22.800766] I [dht-rebalance.c:1117:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-03 03:16:22.815836] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.825731] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/4)
>[2013-12-03 03:16:22.850024] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.859791] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/29)
>[2013-12-03 03:16:22.870339] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.888895] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/30)
>[2013-12-03 03:16:22.898400] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /50: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.899435] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /52: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.900619] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /55: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.901790] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /59: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.903071] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /66: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.907103] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /67: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.916044] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.927814] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/74)
>[2013-12-03 03:16:22.937145] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.947200] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/75)
>[2013-12-03 03:16:22.957070] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:22.972076] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/76)
>[2013-12-03 03:16:22.973929] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /77: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.975078] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /81: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.976430] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /84: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.977760] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /87: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:22.981250] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /90 on hosdu-replicate-2
>[2013-12-03 03:16:22.993932] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2013-12-03 03:16:23.008892] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/90)
>[2013-12-03 03:16:23.025123] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.038697] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/11)
>[2013-12-03 03:16:23.047018] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.060844] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/20)
>[2013-12-03 03:16:23.071069] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.086840] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/35)
>[2013-12-03 03:16:23.094162] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.113757] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/46)
>[2013-12-03 03:16:23.125096] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.141371] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/49)
>[2013-12-03 03:16:23.143652] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /51: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.145327] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /53: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.146741] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /54: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.148289] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /58: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.149796] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /61: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.151048] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /62: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.152323] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /64: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.153448] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /65: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.159681] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.179181] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/68)
>[2013-12-03 03:16:23.193818] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.211401] I [dht-rebalance.c:1776:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1.00 secs
>[2013-12-03 03:16:23.211448] I [dht-rebalance.c:1779:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 63, failures: 18, skipped: 13
>[2013-12-03 03:16:23.223132] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/79)
>[2013-12-03 03:16:23.235851] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.265451] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/85)
>[2013-12-03 03:16:23.275118] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.301962] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/88)
>[2013-12-03 03:16:23.313910] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2013-12-03 03:16:23.356324] W [dht-rebalance.c:374:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-1) with higher disk space to a node (hosdu-replicate-3) with lesser disk space (/93)
>[2013-12-03 03:16:23.358766] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /94: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.360862] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /96: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.362867] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /97: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.364825] E [dht-rebalance.c:1272:gf_defrag_migrate_data] 0-hosdu-dht: /98: failed to get trusted.distribute.linkinfo key - Success
>[2013-12-03 03:16:23.464656] I [dht-rebalance.c:1341:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.66 secs
>[2013-12-03 03:16:23.488591] I [dht-rebalance.c:1776:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2013-12-03 03:16:23.488640] I [dht-rebalance.c:1779:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 22, skipped: 17
>[2013-12-03 03:16:23.489523] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3348ce894d] (-->/lib64/libpthread.so.0() [0x3349407851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1037515: 832062