Attachment 841881 Details for Bug 1046603 – Rebalance logs from peer

Rebalance logs from peer

hosdu-rebalance.log.peer (text/plain), 66.49 KB, created by Lalatendu Mohanty on 2013-12-26 12:12:22 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2013-12-26 12:12:22 UTC

Size: 66.49 KB

patch

obsolete

>[2013-12-25 00:34:22.045645] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=cf334da6-251e-4dd6-9ce9-457081e8feb3 --socket-file /var/lib/glusterd/vols/hosdu/rebalance/cf334da6-251e-4dd6-9ce9-457081e8feb3.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/cf334da6-251e-4dd6-9ce9-457081e8feb3.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 00:34:22.046919] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 00:34:22.052348] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 00:34:22.052390] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 00:34:22.052769] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 00:34:22.052839] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 00:34:27.036444] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'cf334da6-251e-4dd6-9ce9-457081e8feb3'
>[2013-12-25 00:34:27.036490] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 00:34:27.036512] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 00:34:27.036532] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:34:27.036551] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:34:27.036569] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:34:27.036588] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:34:27.036621] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:34:27.036640] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:34:27.036659] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:34:27.036678] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:34:27.036696] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:34:27.036714] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:34:27.036732] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:34:27.036750] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:34:27.036768] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:34:27.036831] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:34:27.036893] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:34:27.036929] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:34:27.036948] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:34:27.036966] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:34:27.036999] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:34:27.037613] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 00:34:27.097445] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 00:34:27.097508] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 00:34:27.100937] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 00:34:27.100956] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 00:34:27.105118] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 00:34:27.105139] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 00:34:27.109452] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 00:34:27.109473] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 00:34:27.114644] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 00:34:27.114666] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 00:34:27.122637] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 00:34:27.122689] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 00:34:27.130035] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 00:34:27.130074] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 00:34:27.135421] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 00:34:27.135441] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 00:34:27.135502] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.140109] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.144306] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.148259] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.152214] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.156037] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.159941] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 00:34:27.164084] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick0
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick1
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick2
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick3
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick4
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick5
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick6
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick7
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid cf334da6-251e-4dd6-9ce9-457081e8feb3
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 00:34:27.171426] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0)
>[2013-12-25 00:34:27.171533] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0)
>[2013-12-25 00:34:27.171628] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2013-12-25 00:34:27.172092] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2013-12-25 00:34:27.172188] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2013-12-25 00:34:27.172278] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0)
>[2013-12-25 00:34:27.196651] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2013-12-25 00:34:27.196770] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2013-12-25 00:34:27.208416] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.208568] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.208693] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.208833] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.209049] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.209871] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.210093] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49152, attached to remote volume '/bricks/hosdu_brick0'.
>[2013-12-25 00:34:27.210111] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.210286] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-25 00:34:27.210331] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.210345] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49152, attached to remote volume '/bricks/hosdu_brick1'.
>[2013-12-25 00:34:27.210355] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.211021] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.211051] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 00:34:27.211160] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.211562] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-3' came back up; going online.
>[2013-12-25 00:34:27.211602] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.211616] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 00:34:27.211626] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.211950] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-5' came back up; going online.
>[2013-12-25 00:34:27.212083] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.212102] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 00:34:27.212111] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.212409] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-25 00:34:27.212446] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.212541] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 00:34:27.212568] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 00:34:27.212587] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 00:34:27.212607] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 00:34:27.212626] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 00:34:27.212646] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.212764] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:34:27.213256] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.213277] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49153, attached to remote volume '/bricks/hosdu_brick7'.
>[2013-12-25 00:34:27.213287] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.213568] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.213588] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 00:34:27.213598] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.214237] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 00:34:27.214268] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 00:34:27.214301] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:34:27.214314] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 00:34:27.214323] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:34:27.215141] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 00:34:27.251443] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 00:34:27.252689] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 00:34:27.252920] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-3: selecting local read_child hosdu-client-7
>[2013-12-25 00:34:27.252959] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 00:34:27.253294] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 00:34:27.254891] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 00:34:27.270454] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 00:34:27.419331] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /90 on hosdu-replicate-2
>[2013-12-25 00:34:27.499282] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /49 on hosdu-replicate-3
>[2013-12-25 00:34:27.548400] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2013-12-25 00:34:27.548474] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 63, failures: 0, skipped: 0
>[2013-12-25 00:34:27.667773] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /18
>[2013-12-25 00:34:27.667855] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /18
>[2013-12-25 00:34:27.811273] W [client-rpc-fops.c:1983:client3_3_setattr_cbk] 0-hosdu-client-7: remote operation failed: No such file or directory
>[2013-12-25 00:34:27.813692] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk] 0-hosdu-client-6: remote operation failed: Success. Path: /86 (f6960519-b03b-4d30-a8b3-0c0f985cf90d)
>[2013-12-25 00:34:27.839333] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.57 secs
>[2013-12-25 00:34:27.883661] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2013-12-25 00:34:27.883724] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 101, failures: 0, skipped: 0
>[2013-12-25 00:34:27.885409] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3dc32e894d] (-->/lib64/libpthread.so.0() [0x3dc3607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down
>[2013-12-25 00:40:44.184705] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=cf334da6-251e-4dd6-9ce9-457081e8feb3 --socket-file /var/lib/glusterd/vols/hosdu/rebalance/cf334da6-251e-4dd6-9ce9-457081e8feb3.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/cf334da6-251e-4dd6-9ce9-457081e8feb3.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 00:40:44.185870] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 00:40:44.191059] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 00:40:44.191101] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 00:40:44.191406] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 00:40:44.191430] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 00:40:49.175233] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'cf334da6-251e-4dd6-9ce9-457081e8feb3'
>[2013-12-25 00:40:49.175257] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 00:40:49.175268] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 00:40:49.175278] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:40:49.175288] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:40:49.175297] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 00:40:49.175307] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:40:49.175323] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:40:49.175333] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:40:49.175342] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 00:40:49.175353] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:40:49.175362] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:40:49.175371] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:40:49.175380] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 00:40:49.175390] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:40:49.175399] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:40:49.175408] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:40:49.175417] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 00:40:49.175427] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:40:49.175436] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:40:49.175445] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:40:49.175467] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 00:40:49.175840] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 00:40:49.227187] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 00:40:49.227268] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 00:40:49.232405] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 00:40:49.232428] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 00:40:49.237866] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 00:40:49.237889] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 00:40:49.247233] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 00:40:49.247265] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 00:40:49.250958] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 00:40:49.250981] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 00:40:49.254624] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 00:40:49.254648] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 00:40:49.257969] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 00:40:49.257992] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 00:40:49.262924] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 00:40:49.262961] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 00:40:49.263029] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.270664] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.275419] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.286383] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.294215] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.298840] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.304410] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 00:40:49.308791] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick2
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick3
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick4
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick5
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick6
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick8
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick9
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick10
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid cf334da6-251e-4dd6-9ce9-457081e8feb3
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 00:40:49.316205] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49153 (from 0)
>[2013-12-25 00:40:49.316329] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49153 (from 0)
>[2013-12-25 00:40:49.316436] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49154 (from 0)
>[2013-12-25 00:40:49.317015] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49154 (from 0)
>[2013-12-25 00:40:49.317124] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49152 (from 0)
>[2013-12-25 00:40:49.317222] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49155 (from 0)
>[2013-12-25 00:40:49.317308] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49156 (from 0)
>[2013-12-25 00:40:49.317568] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49157 (from 0)
>[2013-12-25 00:40:49.368890] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.368977] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.369191] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.369807] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.369829] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49156, attached to remote volume '/bricks/hosdu_brick9'.
>[2013-12-25 00:40:49.369840] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.370050] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-25 00:40:49.370094] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.370107] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49157, attached to remote volume '/bricks/hosdu_brick10'.
>[2013-12-25 00:40:49.370117] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.370826] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.370924] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.370987] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.371031] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.371233] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 00:40:49.371318] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 00:40:49.371343] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 00:40:49.371367] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.371406] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.252:49155, attached to remote volume '/bricks/hosdu_brick8'.
>[2013-12-25 00:40:49.371419] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.371657] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-5' came back up; going online.
>[2013-12-25 00:40:49.371698] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.371711] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 00:40:49.371720] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.371831] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-1' came back up; going online.
>[2013-12-25 00:40:49.371879] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 00:40:49.371904] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.371916] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 00:40:49.371925] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.372395] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.372415] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 00:40:49.372425] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.372525] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-3' came back up; going online.
>[2013-12-25 00:40:49.372585] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.372599] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 00:40:49.372609] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.372816] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 00:40:49.372847] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 00:40:49.372858] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 00:40:49.372868] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 00:40:49.373077] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 00:40:49.373287] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 00:40:49.373369] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 00:40:49.376598] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 00:40:49.399273] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 00:40:49.401505] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 00:40:49.402333] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 00:40:49.403125] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 00:40:49.403452] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 00:40:49.403453] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-3: selecting local read_child hosdu-client-7
>[2013-12-25 00:40:49.405077] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-3: selecting local read_child hosdu-client-6
>[2013-12-25 00:40:49.405931] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-5
>[2013-12-25 00:40:49.407745] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-4
>[2013-12-25 00:40:49.414668] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 00:40:49.599029] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2013-12-25 00:40:49.599169] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 46, failures: 0, skipped: 0
>[2013-12-25 00:40:49.702871] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:49.703861] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:49.703914] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:49.703984] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:50.073529] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /7 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.079328] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.080183] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:50.080274] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:50.080352] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:50.326079] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /18 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.330006] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.330869] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:50.330928] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:50.330979] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:50.651272] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /19 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.660934] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.661831] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:50.661919] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:50.661995] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:50.959077] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /23 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.961572] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:50.962188] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:50.962238] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:50.962282] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:51.225436] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /27 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.228525] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.229509] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:51.229557] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:51.229597] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:51.478321] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /28 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.481335] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.482136] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:51.482185] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:51.482228] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:51.833561] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /32 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.837456] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:51.838055] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:51.838106] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:51.838151] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:52.052268] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /38 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.056686] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.057207] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:52.057257] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:52.057301] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:52.287949] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /44 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.297648] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.298357] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:52.298416] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:52.298469] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:52.569246] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /63 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.574832] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.575459] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:52.575507] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:52.575549] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:52.904998] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /69 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.913078] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:52.914023] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:52.914120] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:52.914193] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:53.122795] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /78 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.126779] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.127315] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:53.127363] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:53.127404] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:53.342540] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /82 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.347859] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.348835] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:53.348930] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:53.349010] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:53.599464] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /86 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.603645] I [dht-rebalance.c:672:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.604147] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x304) [0x7f5722c29e54]))) 0-dict: !this || key=attempt-self-heal
>[2013-12-25 00:40:53.604195] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x329) [0x7f5722c29e79]))) 0-dict: !this || key=foreground-self-heal
>[2013-12-25 00:40:53.604238] W [dict.c:391:dict_del] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/distribute.so(+0x10a09) [0x7f5722994a09] (-->/usr/lib64/libglusterfs.so.0(syncop_lookup+0x1a9) [0x3dc464dc99] (-->/usr/lib64/glusterfs/3.4.0.52rhs/xlator/cluster/replicate.so(afr_lookup+0x34e) [0x7f5722c29e9e]))) 0-dict: !this || key=dry-run-self-heal
>[2013-12-25 00:40:53.832032] I [dht-rebalance.c:881:dht_migrate_file] 0-hosdu-dht: completed migration of /99 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2013-12-25 00:40:53.851046] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 4.44 secs
>[2013-12-25 00:40:53.869913] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 4.00 secs
>[2013-12-25 00:40:53.869961] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 15, size: 196608000, lookups: 115, failures: 0, skipped: 0
>[2013-12-25 00:40:53.870948] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3dc32e894d] (-->/lib64/libpthread.so.0() [0x3dc3607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1046603: 841817 | 841881 | 841882