Attachment 909580 Details for Bug 1110311 – Rebalance logs from the node where commands were executed

Rebalance logs from the node where commands were executed

hosdu-rebalance.log (text/x-log), 56.89 KB, created by Lalatendu Mohanty on 2014-06-17 12:28:26 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2014-06-17 12:28:26 UTC

Size: 56.89 KB

patch

obsolete

>[2014-06-16 20:40:23.036411] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.17 (args: /usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=e1ecdfdd-114d-40af-8b02-78fe44595c47 --socket-file /var/run/gluster/gluster-rebalance-c38f85bc-c244-4e5d-a101-7e0a9965f909.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/e1ecdfdd-114d-40af-8b02-78fe44595c47.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2014-06-16 20:40:28.038155] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'e1ecdfdd-114d-40af-8b02-78fe44595c47'
>[2014-06-16 20:40:28.038175] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2014-06-16 20:40:28.038186] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2014-06-16 20:40:28.038193] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:40:28.038200] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:40:28.038206] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:40:28.038214] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:40:28.038223] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:40:28.038233] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:40:28.038239] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:40:28.038247] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:40:28.038253] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:40:28.038259] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:40:28.038265] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:40:28.038273] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:40:28.038279] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:40:28.038285] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:40:28.038291] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:40:28.038298] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:40:28.038304] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:40:28.038316] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:40:28.038326] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:40:28.038817] I [dht-shared.c:332:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-06-16 20:40:28.044121] I [client.c:2280:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.047985] I [client.c:2280:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.051069] I [client.c:2280:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.054004] I [client.c:2280:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.056885] I [client.c:2280:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.059897] I [client.c:2280:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.062996] I [client.c:2280:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2014-06-16 20:40:28.066076] I [client.c:2280:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host rhsauto056.lab.eng.blr.redhat.com
>  5:     option remote-subvolume /bricks/hosdu_brick0
>  6:     option transport-type socket
>  7:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>  8:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume hosdu-client-1
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host rhsauto057.lab.eng.blr.redhat.com
> 16:     option remote-subvolume /bricks/hosdu_brick1
> 17:     option transport-type socket
> 18:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 19:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume hosdu-replicate-0
> 24:     type cluster/replicate
> 25:     option data-self-heal off
> 26:     option metadata-self-heal off
> 27:     option entry-self-heal off
> 28:     option readdir-failover off
> 29:     subvolumes hosdu-client-0 hosdu-client-1
> 30: end-volume
> 31:  
> 32: volume hosdu-client-2
> 33:     type protocol/client
> 34:     option ping-timeout 42
> 35:     option remote-host rhsauto056.lab.eng.blr.redhat.com
> 36:     option remote-subvolume /bricks/hosdu_brick2
> 37:     option transport-type socket
> 38:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 39:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 40:     option send-gids true
> 41: end-volume
> 42:  
> 43: volume hosdu-client-3
> 44:     type protocol/client
> 45:     option ping-timeout 42
> 46:     option remote-host rhsauto057.lab.eng.blr.redhat.com
> 47:     option remote-subvolume /bricks/hosdu_brick3
> 48:     option transport-type socket
> 49:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 50:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 51:     option send-gids true
> 52: end-volume
> 53:  
> 54: volume hosdu-replicate-1
> 55:     type cluster/replicate
> 56:     option data-self-heal off
> 57:     option metadata-self-heal off
> 58:     option entry-self-heal off
> 59:     option readdir-failover off
> 60:     subvolumes hosdu-client-2 hosdu-client-3
> 61: end-volume
> 62:  
> 63: volume hosdu-client-4
> 64:     type protocol/client
> 65:     option ping-timeout 42
> 66:     option remote-host rhsauto056.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick4
> 68:     option transport-type socket
> 69:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 70:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 71:     option send-gids true
> 72: end-volume
> 73:  
> 74: volume hosdu-client-5
> 75:     type protocol/client
> 76:     option ping-timeout 42
> 77:     option remote-host rhsauto057.lab.eng.blr.redhat.com
> 78:     option remote-subvolume /bricks/hosdu_brick5
> 79:     option transport-type socket
> 80:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 81:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 82:     option send-gids true
> 83: end-volume
> 84:  
> 85: volume hosdu-replicate-2
> 86:     type cluster/replicate
> 87:     option data-self-heal off
> 88:     option metadata-self-heal off
> 89:     option entry-self-heal off
> 90:     option readdir-failover off
> 91:     subvolumes hosdu-client-4 hosdu-client-5
> 92: end-volume
> 93:  
> 94: volume hosdu-client-6
> 95:     type protocol/client
> 96:     option ping-timeout 42
> 97:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 98:     option remote-subvolume /bricks/hosdu_brick6
> 99:     option transport-type socket
>100:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>101:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>102:     option send-gids true
>103: end-volume
>104:  
>105: volume hosdu-client-7
>106:     type protocol/client
>107:     option ping-timeout 42
>108:     option remote-host rhsauto022.lab.eng.blr.redhat.com
>109:     option remote-subvolume /bricks/hosdu_brick7
>110:     option transport-type socket
>111:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>112:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>113:     option send-gids true
>114: end-volume
>115:  
>116: volume hosdu-replicate-3
>117:     type cluster/replicate
>118:     option data-self-heal off
>119:     option metadata-self-heal off
>120:     option entry-self-heal off
>121:     option readdir-failover off
>122:     subvolumes hosdu-client-6 hosdu-client-7
>123: end-volume
>124:  
>125: volume hosdu-dht
>126:     type cluster/distribute
>127:     option use-readdirp yes
>128:     option lookup-unhashed yes
>129:     option assert-no-child-down yes
>130:     option readdir-optimize on
>131:     option rebalance-cmd 1
>132:     option node-uuid e1ecdfdd-114d-40af-8b02-78fe44595c47
>133:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>134: end-volume
>135:  
>136: volume hosdu-write-behind
>137:     type performance/write-behind
>138:     subvolumes hosdu-dht
>139: end-volume
>140:  
>141: volume hosdu-read-ahead
>142:     type performance/read-ahead
>143:     subvolumes hosdu-write-behind
>144: end-volume
>145:  
>146: volume hosdu-readdir-ahead
>147:     type performance/readdir-ahead
>148:     subvolumes hosdu-read-ahead
>149: end-volume
>150:  
>151: volume hosdu-io-cache
>152:     type performance/io-cache
>153:     subvolumes hosdu-readdir-ahead
>154: end-volume
>155:  
>156: volume hosdu-quick-read
>157:     type performance/quick-read
>158:     subvolumes hosdu-io-cache
>159: end-volume
>160:  
>161: volume hosdu-md-cache
>162:     type performance/md-cache
>163:     subvolumes hosdu-quick-read
>164: end-volume
>165:  
>166: volume hosdu
>167:     type debug/io-stats
>168:     option latency-measurement on
>169:     option count-fop-hits on
>170:     subvolumes hosdu-md-cache
>171: end-volume
>172:  
>+------------------------------------------------------------------------------+
>[2014-06-16 20:40:28.071117] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0)
>[2014-06-16 20:40:28.071180] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2014-06-16 20:40:28.071228] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2014-06-16 20:40:28.081155] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:28.081360] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:28.081570] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-0: Connected to hosdu-client-0, attached to remote volume '/bricks/hosdu_brick0'.
>[2014-06-16 20:40:28.081584] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:28.081720] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2014-06-16 20:40:28.081756] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-2: Connected to hosdu-client-2, attached to remote volume '/bricks/hosdu_brick2'.
>[2014-06-16 20:40:28.081764] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:28.081878] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online.
>[2014-06-16 20:40:28.081903] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:28.081959] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2014-06-16 20:40:28.081975] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2014-06-16 20:40:28.082249] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-4: Connected to hosdu-client-4, attached to remote volume '/bricks/hosdu_brick4'.
>[2014-06-16 20:40:28.082268] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:28.082344] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online.
>[2014-06-16 20:40:28.082409] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2014-06-16 20:40:33.048735] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2014-06-16 20:40:33.052080] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0)
>[2014-06-16 20:40:33.056623] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0)
>[2014-06-16 20:40:33.060297] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2014-06-16 20:40:33.060373] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2014-06-16 20:40:33.060466] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:33.066588] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:33.066887] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-6: Connected to hosdu-client-6, attached to remote volume '/bricks/hosdu_brick6'.
>[2014-06-16 20:40:33.066902] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:33.067102] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2014-06-16 20:40:33.067375] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:33.067796] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:33.067956] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:40:33.068124] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2014-06-16 20:40:33.068153] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-7: Connected to hosdu-client-7, attached to remote volume '/bricks/hosdu_brick7'.
>[2014-06-16 20:40:33.068160] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:33.068747] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-3: Connected to hosdu-client-3, attached to remote volume '/bricks/hosdu_brick3'.
>[2014-06-16 20:40:33.068761] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:33.069444] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-5: Connected to hosdu-client-5, attached to remote volume '/bricks/hosdu_brick5'.
>[2014-06-16 20:40:33.069469] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:33.070007] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-1: Connected to hosdu-client-1, attached to remote volume '/bricks/hosdu_brick1'.
>[2014-06-16 20:40:33.070032] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:40:33.070629] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2014-06-16 20:40:33.070649] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2014-06-16 20:40:33.070672] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2014-06-16 20:40:33.070684] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2014-06-16 20:40:33.079028] I [dht-common.c:2601:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2014-06-16 20:40:33.085514] I [dht-rebalance.c:1220:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2014-06-16 20:40:33.096861] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.106598] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/4)
>[2014-06-16 20:40:33.120819] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.130055] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/29)
>[2014-06-16 20:40:33.136832] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.144755] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/30)
>[2014-06-16 20:40:33.169065] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.180017] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/74)
>[2014-06-16 20:40:33.192404] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.209926] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/75)
>[2014-06-16 20:40:33.229640] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.246914] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/76)
>[2014-06-16 20:40:33.276267] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:40:33.300827] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2014-06-16 20:40:33.300853] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 33, failures: 0, skipped: 6
>[2014-06-16 20:40:33.306708] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/90)
>[2014-06-16 20:40:33.322924] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:34.544584] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /11 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:34.554619] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:35.583531] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /20 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:35.596305] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:36.617071] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /35 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:36.623579] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:37.745135] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /46 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:37.753705] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:38.680644] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 5.00 secs
>[2014-06-16 20:40:38.680668] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 52428800, lookups: 50, failures: 0, skipped: 7
>[2014-06-16 20:40:38.829285] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /49 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:38.849063] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:39.897260] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /68 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:39.906865] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:40.978393] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /79 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:40.984651] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:41.994406] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /85 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:42.000420] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:43.027318] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /88 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:43.035060] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:43.986608] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 10.00 secs
>[2014-06-16 20:40:43.986629] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 117964800, lookups: 67, failures: 0, skipped: 7
>[2014-06-16 20:40:44.062702] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /93 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:40:44.074947] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:45.102352] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /7 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:45.109806] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:46.120359] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /18 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:46.126900] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:47.171865] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /19 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:47.178599] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:48.573681] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /23 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:48.582318] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:49.299474] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 16.00 secs
>[2014-06-16 20:40:49.299522] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 183500800, lookups: 78, failures: 0, skipped: 7
>[2014-06-16 20:40:50.485945] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /27 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:50.495583] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:52.323162] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /28 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:52.333970] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:54.222627] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /32 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:54.236954] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:54.863089] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 21.00 secs
>[2014-06-16 20:40:54.863138] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 222822400, lookups: 83, failures: 0, skipped: 7
>[2014-06-16 20:40:56.266804] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /38 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:56.282481] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:58.069031] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /44 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:40:58.085980] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:00.047313] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /63 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:00.055654] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:00.565354] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 27.00 secs
>[2014-06-16 20:41:00.565401] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 262144000, lookups: 91, failures: 0, skipped: 7
>[2014-06-16 20:41:01.952688] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /69 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:01.963585] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:03.885819] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /78 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:03.898533] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:05.720151] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 32.00 secs
>[2014-06-16 20:41:05.720175] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 288358400, lookups: 95, failures: 0, skipped: 7
>[2014-06-16 20:41:05.875591] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /82 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:05.887232] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:07.523698] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /86 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:07.533103] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:08.655789] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /99 from subvolume hosdu-replicate-2 to hosdu-replicate-3
>[2014-06-16 20:41:08.729540] I [dht-rebalance.c:1443:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 35.64 secs
>[2014-06-16 20:41:08.739544] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 35.00 secs
>[2014-06-16 20:41:08.739564] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 25, size: 327680000, lookups: 125, failures: 0, skipped: 7
>[2014-06-16 20:41:08.740108] W [glusterfsd.c:1182:cleanup_and_exit] (--> 0-: received signum (15), shutting down
>[2014-06-16 20:48:05.729605] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.17 (args: /usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=e1ecdfdd-114d-40af-8b02-78fe44595c47 --socket-file /var/run/gluster/gluster-rebalance-c38f85bc-c244-4e5d-a101-7e0a9965f909.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/e1ecdfdd-114d-40af-8b02-78fe44595c47.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2014-06-16 20:48:10.727378] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'e1ecdfdd-114d-40af-8b02-78fe44595c47'
>[2014-06-16 20:48:10.727406] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2014-06-16 20:48:10.727419] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2014-06-16 20:48:10.727428] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:48:10.727435] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:48:10.727443] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2014-06-16 20:48:10.727450] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:48:10.727460] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:48:10.727471] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:48:10.727478] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2014-06-16 20:48:10.727487] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:48:10.727494] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:48:10.727500] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:48:10.727507] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2014-06-16 20:48:10.727515] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:48:10.727522] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:48:10.727529] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:48:10.727536] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2014-06-16 20:48:10.727543] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:48:10.727550] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:48:10.727557] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:48:10.727563] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2014-06-16 20:48:10.728102] I [dht-shared.c:332:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-06-16 20:48:10.734090] I [client.c:2280:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.738410] I [client.c:2280:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.741613] I [client.c:2280:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.744618] I [client.c:2280:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.747801] I [client.c:2280:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.750806] I [client.c:2280:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.754126] I [client.c:2280:notify] 0-hosdu-client-8: parent translators are ready, attempting connect on transport
>[2014-06-16 20:48:10.757232] I [client.c:2280:notify] 0-hosdu-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-2
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host rhsauto056.lab.eng.blr.redhat.com
>  5:     option remote-subvolume /bricks/hosdu_brick2
>  6:     option transport-type socket
>  7:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>  8:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume hosdu-client-3
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host rhsauto057.lab.eng.blr.redhat.com
> 16:     option remote-subvolume /bricks/hosdu_brick3
> 17:     option transport-type socket
> 18:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 19:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume hosdu-replicate-0
> 24:     type cluster/replicate
> 25:     option data-self-heal off
> 26:     option metadata-self-heal off
> 27:     option entry-self-heal off
> 28:     option readdir-failover off
> 29:     subvolumes hosdu-client-2 hosdu-client-3
> 30: end-volume
> 31:  
> 32: volume hosdu-client-4
> 33:     type protocol/client
> 34:     option ping-timeout 42
> 35:     option remote-host rhsauto056.lab.eng.blr.redhat.com
> 36:     option remote-subvolume /bricks/hosdu_brick4
> 37:     option transport-type socket
> 38:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 39:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 40:     option send-gids true
> 41: end-volume
> 42:  
> 43: volume hosdu-client-5
> 44:     type protocol/client
> 45:     option ping-timeout 42
> 46:     option remote-host rhsauto057.lab.eng.blr.redhat.com
> 47:     option remote-subvolume /bricks/hosdu_brick5
> 48:     option transport-type socket
> 49:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 50:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 51:     option send-gids true
> 52: end-volume
> 53:  
> 54: volume hosdu-replicate-1
> 55:     type cluster/replicate
> 56:     option data-self-heal off
> 57:     option metadata-self-heal off
> 58:     option entry-self-heal off
> 59:     option readdir-failover off
> 60:     subvolumes hosdu-client-4 hosdu-client-5
> 61: end-volume
> 62:  
> 63: volume hosdu-client-6
> 64:     type protocol/client
> 65:     option ping-timeout 42
> 66:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick6
> 68:     option transport-type socket
> 69:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 70:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 71:     option send-gids true
> 72: end-volume
> 73:  
> 74: volume hosdu-client-7
> 75:     type protocol/client
> 76:     option ping-timeout 42
> 77:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 78:     option remote-subvolume /bricks/hosdu_brick8
> 79:     option transport-type socket
> 80:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
> 81:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
> 82:     option send-gids true
> 83: end-volume
> 84:  
> 85: volume hosdu-replicate-2
> 86:     type cluster/replicate
> 87:     option data-self-heal off
> 88:     option metadata-self-heal off
> 89:     option entry-self-heal off
> 90:     option readdir-failover off
> 91:     subvolumes hosdu-client-6 hosdu-client-7
> 92: end-volume
> 93:  
> 94: volume hosdu-client-8
> 95:     type protocol/client
> 96:     option ping-timeout 42
> 97:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 98:     option remote-subvolume /bricks/hosdu_brick9
> 99:     option transport-type socket
>100:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>101:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>102:     option send-gids true
>103: end-volume
>104:  
>105: volume hosdu-client-9
>106:     type protocol/client
>107:     option ping-timeout 42
>108:     option remote-host rhsauto022.lab.eng.blr.redhat.com
>109:     option remote-subvolume /bricks/hosdu_brick10
>110:     option transport-type socket
>111:     option username e61519f5-94f4-4e5b-b0ee-b55bebe09060
>112:     option password d467cbcc-4275-45fb-988e-7bcd55923c21
>113:     option send-gids true
>114: end-volume
>115:  
>116: volume hosdu-replicate-3
>117:     type cluster/replicate
>118:     option data-self-heal off
>119:     option metadata-self-heal off
>120:     option entry-self-heal off
>121:     option readdir-failover off
>122:     subvolumes hosdu-client-8 hosdu-client-9
>123: end-volume
>124:  
>125: volume hosdu-dht
>126:     type cluster/distribute
>127:     option use-readdirp yes
>128:     option lookup-unhashed yes
>129:     option assert-no-child-down yes
>130:     option readdir-optimize on
>131:     option rebalance-cmd 1
>132:     option node-uuid e1ecdfdd-114d-40af-8b02-78fe44595c47
>133:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>134: end-volume
>135:  
>136: volume hosdu-write-behind
>137:     type performance/write-behind
>138:     subvolumes hosdu-dht
>139: end-volume
>140:  
>141: volume hosdu-read-ahead
>142:     type performance/read-ahead
>143:     subvolumes hosdu-write-behind
>144: end-volume
>145:  
>146: volume hosdu-readdir-ahead
>147:     type performance/readdir-ahead
>148:     subvolumes hosdu-read-ahead
>149: end-volume
>150:  
>151: volume hosdu-io-cache
>152:     type performance/io-cache
>153:     subvolumes hosdu-readdir-ahead
>154: end-volume
>155:  
>156: volume hosdu-quick-read
>157:     type performance/quick-read
>158:     subvolumes hosdu-io-cache
>159: end-volume
>160:  
>161: volume hosdu-md-cache
>162:     type performance/md-cache
>163:     subvolumes hosdu-quick-read
>164: end-volume
>165:  
>166: volume hosdu
>167:     type debug/io-stats
>168:     option latency-measurement on
>169:     option count-fop-hits on
>170:     subvolumes hosdu-md-cache
>171: end-volume
>172:  
>+------------------------------------------------------------------------------+
>[2014-06-16 20:48:10.762241] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2014-06-16 20:48:10.765694] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2014-06-16 20:48:10.769198] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:10.769432] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-4: Connected to hosdu-client-4, attached to remote volume '/bricks/hosdu_brick4'.
>[2014-06-16 20:48:10.769446] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:10.769569] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-4' came back up; going online.
>[2014-06-16 20:48:10.769610] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:10.769672] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2014-06-16 20:48:10.770135] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-2: Connected to hosdu-client-2, attached to remote volume '/bricks/hosdu_brick2'.
>[2014-06-16 20:48:10.770149] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:10.770206] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-2' came back up; going online.
>[2014-06-16 20:48:10.770363] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2014-06-16 20:48:15.743862] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2014-06-16 20:48:15.748023] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49155 (from 0)
>[2014-06-16 20:48:15.748166] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-8: changing port to 49156 (from 0)
>[2014-06-16 20:48:15.748217] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-9: changing port to 49157 (from 0)
>[2014-06-16 20:48:15.770315] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2014-06-16 20:48:15.778869] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2014-06-16 20:48:15.780289] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.788192] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.788425] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.788892] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-6: Connected to hosdu-client-6, attached to remote volume '/bricks/hosdu_brick6'.
>[2014-06-16 20:48:15.788926] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.789745] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-6' came back up; going online.
>[2014-06-16 20:48:15.789814] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.790371] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2014-06-16 20:48:15.790429] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-7: Connected to hosdu-client-7, attached to remote volume '/bricks/hosdu_brick8'.
>[2014-06-16 20:48:15.790449] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.791652] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.792135] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-8: Connected to hosdu-client-8, attached to remote volume '/bricks/hosdu_brick9'.
>[2014-06-16 20:48:15.792166] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.792580] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-8' came back up; going online.
>[2014-06-16 20:48:15.792860] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2014-06-16 20:48:15.792912] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-3: Connected to hosdu-client-3, attached to remote volume '/bricks/hosdu_brick3'.
>[2014-06-16 20:48:15.792931] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.793766] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-9: Connected to hosdu-client-9, attached to remote volume '/bricks/hosdu_brick10'.
>[2014-06-16 20:48:15.793937] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.794890] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-06-16 20:48:15.795296] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2014-06-16 20:48:15.795489] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-9: Server lk version = 1
>[2014-06-16 20:48:15.795534] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-8: Server lk version = 1
>[2014-06-16 20:48:15.795960] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-5: Connected to hosdu-client-5, attached to remote volume '/bricks/hosdu_brick5'.
>[2014-06-16 20:48:15.795989] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-06-16 20:48:15.797789] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2014-06-16 20:48:15.814794] I [dht-common.c:2601:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2014-06-16 20:48:15.828918] I [dht-rebalance.c:1220:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2014-06-16 20:48:15.852763] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:15.987891] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2014-06-16 20:48:15.996384] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-06-16 20:48:18.217334] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /4 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:18.238902] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:19.926325] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /29 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:19.932097] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:21.379081] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /30 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:21.422356] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:21.549950] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 6.00 secs
>[2014-06-16 20:48:21.549993] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 39321600, lookups: 26, failures: 0, skipped: 0
>[2014-06-16 20:48:23.630311] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /74 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:23.640393] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:25.710154] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /75 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:25.720831] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:26.889392] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 11.00 secs
>[2014-06-16 20:48:26.889449] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 5, size: 65536000, lookups: 28, failures: 0, skipped: 0
>[2014-06-16 20:48:27.579995] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /76 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:27.598599] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:29.269825] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /90 from subvolume hosdu-replicate-0 to hosdu-replicate-2
>[2014-06-16 20:48:29.299375] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:31.195233] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /11 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:31.208639] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:32.211654] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 17.00 secs
>[2014-06-16 20:48:32.211701] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 8, size: 104857600, lookups: 42, failures: 0, skipped: 0
>[2014-06-16 20:48:33.257841] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /20 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:33.276720] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:35.454971] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /35 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:35.461837] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:37.327490] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /46 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:37.340118] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:37.562564] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 22.00 secs
>[2014-06-16 20:48:37.562657] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 11, size: 144179200, lookups: 50, failures: 0, skipped: 0
>[2014-06-16 20:48:39.043848] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /49 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:39.067002] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:40.604011] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /68 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:40.619525] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:42.627333] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /79 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:42.637902] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:42.884400] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 27.00 secs
>[2014-06-16 20:48:42.884447] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 183500800, lookups: 64, failures: 0, skipped: 0
>[2014-06-16 20:48:44.668533] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /85 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:44.679070] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:46.733815] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /88 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:46.743083] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:48.197683] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 33.00 secs
>[2014-06-16 20:48:48.197760] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 209715200, lookups: 67, failures: 0, skipped: 0
>[2014-06-16 20:48:48.617216] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /93 from subvolume hosdu-replicate-1 to hosdu-replicate-3
>[2014-06-16 20:48:48.812273] I [dht-rebalance.c:1443:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 32.98 secs
>[2014-06-16 20:48:48.840375] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 33.00 secs
>[2014-06-16 20:48:48.840430] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 222822400, lookups: 117, failures: 0, skipped: 0
>[2014-06-16 20:48:48.841593] W [glusterfsd.c:1182:cleanup_and_exit] (--> 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1110311: 909580 | 909582 | 911310 | 913381