Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 909580 Details for
Bug 1110311
BVT: Rebalance status reporting failure
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Rebalance logs from the node where commands were executed
hosdu-rebalance.log (text/x-log), 56.89 KB, created by
Lalatendu Mohanty
on 2014-06-17 12:28:26 UTC
(
hide
)
Description:
Rebalance logs from the node where commands were executed
Filename:
MIME Type:
Creator:
Lalatendu Mohanty
Created:
2014-06-17 12:28:26 UTC
Size:
56.89 KB
patch
obsolete
>[2014-06-16 20:40:23.036411] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.17 (args: /usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=e1ecdfdd-114d-40af-8b02-78fe44595c47 --socket-file /var/run/gluster/gluster-rebalance-c38f85bc-c244-4e5d-a101-7e0a9965f909.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/e1ecdfdd-114d-40af-8b02-78fe44595c47.pid -l /var/log/glusterfs/hosdu-rebalance.log) >[2014-06-16 20:40:28.038155] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'e1ecdfdd-114d-40af-8b02-78fe44595c47' >[2014-06-16 20:40:28.038175] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1' >[2014-06-16 20:40:28.038186] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on' >[2014-06-16 20:40:28.038193] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:40:28.038200] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:40:28.038206] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:40:28.038214] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:40:28.038223] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:40:28.038233] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:40:28.038239] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:40:28.038247] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:40:28.038253] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:40:28.038259] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:40:28.038265] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:40:28.038273] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:40:28.038279] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:40:28.038285] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:40:28.038291] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:40:28.038298] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:40:28.038304] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:40:28.038316] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:40:28.038326] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:40:28.038817] I [dht-shared.c:332:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-06-16 20:40:28.044121] I [client.c:2280:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.047985] I [client.c:2280:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.051069] I [client.c:2280:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.054004] I [client.c:2280:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.056885] I [client.c:2280:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.059897] I [client.c:2280:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.062996] I [client.c:2280:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport >[2014-06-16 20:40:28.066076] I [client.c:2280:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume hosdu-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host rhsauto056.lab.eng.blr.redhat.com > 5: option remote-subvolume /bricks/hosdu_brick0 > 6: option transport-type socket > 7: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 8: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 9: option send-gids true > 10: end-volume > 11: > 12: volume hosdu-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host rhsauto057.lab.eng.blr.redhat.com > 16: option remote-subvolume /bricks/hosdu_brick1 > 17: option transport-type socket > 18: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 19: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 20: option send-gids true > 21: end-volume > 22: > 23: volume hosdu-replicate-0 > 24: type cluster/replicate > 25: option data-self-heal off > 26: option metadata-self-heal off > 27: option entry-self-heal off > 28: option readdir-failover off > 29: subvolumes hosdu-client-0 hosdu-client-1 > 30: end-volume > 31: > 32: volume hosdu-client-2 > 33: type protocol/client > 34: option ping-timeout 42 > 35: option remote-host rhsauto056.lab.eng.blr.redhat.com > 36: option remote-subvolume /bricks/hosdu_brick2 > 37: option transport-type socket > 38: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 39: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 40: option send-gids true > 41: end-volume > 42: > 43: volume hosdu-client-3 > 44: type protocol/client > 45: option ping-timeout 42 > 46: option remote-host rhsauto057.lab.eng.blr.redhat.com > 47: option remote-subvolume /bricks/hosdu_brick3 > 48: option transport-type socket > 49: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 50: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 51: option send-gids true > 52: end-volume > 53: > 54: volume hosdu-replicate-1 > 55: type cluster/replicate > 56: option data-self-heal off > 57: option metadata-self-heal off > 58: option entry-self-heal off > 59: option readdir-failover off > 60: subvolumes hosdu-client-2 hosdu-client-3 > 61: end-volume > 62: > 63: volume hosdu-client-4 > 64: type protocol/client > 65: option ping-timeout 42 > 66: option remote-host rhsauto056.lab.eng.blr.redhat.com > 67: option remote-subvolume /bricks/hosdu_brick4 > 68: option transport-type socket > 69: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 70: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 71: option send-gids true > 72: end-volume > 73: > 74: volume hosdu-client-5 > 75: type protocol/client > 76: option ping-timeout 42 > 77: option remote-host rhsauto057.lab.eng.blr.redhat.com > 78: option remote-subvolume /bricks/hosdu_brick5 > 79: option transport-type socket > 80: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 81: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 82: option send-gids true > 83: end-volume > 84: > 85: volume hosdu-replicate-2 > 86: type cluster/replicate > 87: option data-self-heal off > 88: option metadata-self-heal off > 89: option entry-self-heal off > 90: option readdir-failover off > 91: subvolumes hosdu-client-4 hosdu-client-5 > 92: end-volume > 93: > 94: volume hosdu-client-6 > 95: type protocol/client > 96: option ping-timeout 42 > 97: option remote-host rhsauto022.lab.eng.blr.redhat.com > 98: option remote-subvolume /bricks/hosdu_brick6 > 99: option transport-type socket >100: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 >101: option password d467cbcc-4275-45fb-988e-7bcd55923c21 >102: option send-gids true >103: end-volume >104: >105: volume hosdu-client-7 >106: type protocol/client >107: option ping-timeout 42 >108: option remote-host rhsauto022.lab.eng.blr.redhat.com >109: option remote-subvolume /bricks/hosdu_brick7 >110: option transport-type socket >111: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 >112: option password d467cbcc-4275-45fb-988e-7bcd55923c21 >113: option send-gids true >114: end-volume >115: >116: volume hosdu-replicate-3 >117: type cluster/replicate >118: option data-self-heal off >119: option metadata-self-heal off >120: option entry-self-heal off >121: option readdir-failover off >122: subvolumes hosdu-client-6 hosdu-client-7 >123: end-volume >124: >125: volume hosdu-dht >126: type cluster/distribute >127: option use-readdirp yes >128: option lookup-unhashed yes >129: option assert-no-child-down yes >130: option readdir-optimize on >131: option rebalance-cmd 1 >132: option node-uuid e1ecdfdd-114d-40af-8b02-78fe44595c47 >133: subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3 >134: end-volume >135: >136: volume hosdu-write-behind >137: type performance/write-behind >138: subvolumes hosdu-dht >139: end-volume >140: >141: volume hosdu-read-ahead >142: type performance/read-ahead >143: subvolumes hosdu-write-behind >144: end-volume >145: >146: volume hosdu-readdir-ahead >147: type performance/readdir-ahead >148: subvolumes hosdu-read-ahead >149: end-volume >150: >151: volume hosdu-io-cache >152: type performance/io-cache >153: subvolumes hosdu-readdir-ahead >154: end-volume >155: >156: volume hosdu-quick-read >157: type performance/quick-read >158: subvolumes hosdu-io-cache >159: end-volume >160: >161: volume hosdu-md-cache >162: type performance/md-cache >163: subvolumes hosdu-quick-read >164: end-volume >165: >166: volume hosdu >167: type debug/io-stats >168: option latency-measurement on >169: option count-fop-hits on >170: subvolumes hosdu-md-cache >171: end-volume >172: >+------------------------------------------------------------------------------+ >[2014-06-16 20:40:28.071117] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0) >[2014-06-16 20:40:28.071180] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0) >[2014-06-16 20:40:28.071228] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0) >[2014-06-16 20:40:28.081155] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:28.081360] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:28.081570] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-0: Connected to hosdu-client-0, attached to remote volume '/bricks/hosdu_brick0'. >[2014-06-16 20:40:28.081584] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:28.081720] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online. >[2014-06-16 20:40:28.081756] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-2: Connected to hosdu-client-2, attached to remote volume '/bricks/hosdu_brick2'. >[2014-06-16 20:40:28.081764] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:28.081878] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-2' came back up; going online. >[2014-06-16 20:40:28.081903] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:28.081959] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1 >[2014-06-16 20:40:28.081975] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1 >[2014-06-16 20:40:28.082249] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-4: Connected to hosdu-client-4, attached to remote volume '/bricks/hosdu_brick4'. >[2014-06-16 20:40:28.082268] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:28.082344] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online. >[2014-06-16 20:40:28.082409] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1 >[2014-06-16 20:40:33.048735] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0) >[2014-06-16 20:40:33.052080] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0) >[2014-06-16 20:40:33.056623] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0) >[2014-06-16 20:40:33.060297] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0) >[2014-06-16 20:40:33.060373] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0) >[2014-06-16 20:40:33.060466] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:33.066588] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:33.066887] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-6: Connected to hosdu-client-6, attached to remote volume '/bricks/hosdu_brick6'. >[2014-06-16 20:40:33.066902] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:33.067102] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online. >[2014-06-16 20:40:33.067375] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:33.067796] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:33.067956] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:40:33.068124] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1 >[2014-06-16 20:40:33.068153] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-7: Connected to hosdu-client-7, attached to remote volume '/bricks/hosdu_brick7'. >[2014-06-16 20:40:33.068160] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:33.068747] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-3: Connected to hosdu-client-3, attached to remote volume '/bricks/hosdu_brick3'. >[2014-06-16 20:40:33.068761] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:33.069444] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-5: Connected to hosdu-client-5, attached to remote volume '/bricks/hosdu_brick5'. >[2014-06-16 20:40:33.069469] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:33.070007] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-1: Connected to hosdu-client-1, attached to remote volume '/bricks/hosdu_brick1'. >[2014-06-16 20:40:33.070032] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:40:33.070629] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1 >[2014-06-16 20:40:33.070649] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1 >[2014-06-16 20:40:33.070672] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1 >[2014-06-16 20:40:33.070684] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1 >[2014-06-16 20:40:33.079028] I [dht-common.c:2601:dht_setxattr] 0-hosdu-dht: fixing the layout of / >[2014-06-16 20:40:33.085514] I [dht-rebalance.c:1220:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on / >[2014-06-16 20:40:33.096861] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.106598] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/4) >[2014-06-16 20:40:33.120819] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.130055] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/29) >[2014-06-16 20:40:33.136832] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.144755] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/30) >[2014-06-16 20:40:33.169065] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.180017] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/74) >[2014-06-16 20:40:33.192404] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.209926] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/75) >[2014-06-16 20:40:33.229640] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.246914] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/76) >[2014-06-16 20:40:33.276267] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:40:33.300827] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs >[2014-06-16 20:40:33.300853] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 33, failures: 0, skipped: 6 >[2014-06-16 20:40:33.306708] W [dht-rebalance.c:453:__dht_check_free_space] 0-hosdu-dht: data movement attempted from node (hosdu-replicate-0) with higher disk space to a node (hosdu-replicate-2) with lesser disk space (/90) >[2014-06-16 20:40:33.322924] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:34.544584] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /11 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:34.554619] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:35.583531] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /20 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:35.596305] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:36.617071] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /35 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:36.623579] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:37.745135] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /46 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:37.753705] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:38.680644] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 5.00 secs >[2014-06-16 20:40:38.680668] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 4, size: 52428800, lookups: 50, failures: 0, skipped: 7 >[2014-06-16 20:40:38.829285] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /49 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:38.849063] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:39.897260] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /68 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:39.906865] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:40.978393] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /79 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:40.984651] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:41.994406] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /85 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:42.000420] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:43.027318] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /88 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:43.035060] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:43.986608] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 10.00 secs >[2014-06-16 20:40:43.986629] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 9, size: 117964800, lookups: 67, failures: 0, skipped: 7 >[2014-06-16 20:40:44.062702] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /93 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:40:44.074947] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /7: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:45.102352] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /7 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:45.109806] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /18: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:46.120359] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /18 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:46.126900] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /19: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:47.171865] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /19 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:47.178599] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /23: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:48.573681] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /23 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:48.582318] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /27: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:49.299474] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 16.00 secs >[2014-06-16 20:40:49.299522] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 183500800, lookups: 78, failures: 0, skipped: 7 >[2014-06-16 20:40:50.485945] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /27 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:50.495583] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /28: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:52.323162] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /28 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:52.333970] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /32: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:54.222627] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /32 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:54.236954] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /38: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:54.863089] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 21.00 secs >[2014-06-16 20:40:54.863138] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 222822400, lookups: 83, failures: 0, skipped: 7 >[2014-06-16 20:40:56.266804] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /38 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:56.282481] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /44: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:58.069031] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /44 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:40:58.085980] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /63: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:00.047313] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /63 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:00.055654] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /69: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:00.565354] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 27.00 secs >[2014-06-16 20:41:00.565401] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 20, size: 262144000, lookups: 91, failures: 0, skipped: 7 >[2014-06-16 20:41:01.952688] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /69 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:01.963585] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /78: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:03.885819] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /78 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:03.898533] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /82: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:05.720151] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 32.00 secs >[2014-06-16 20:41:05.720175] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 22, size: 288358400, lookups: 95, failures: 0, skipped: 7 >[2014-06-16 20:41:05.875591] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /82 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:05.887232] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /86: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:07.523698] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /86 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:07.533103] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /99: attempting to move from hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:08.655789] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /99 from subvolume hosdu-replicate-2 to hosdu-replicate-3 >[2014-06-16 20:41:08.729540] I [dht-rebalance.c:1443:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 35.64 secs >[2014-06-16 20:41:08.739544] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 35.00 secs >[2014-06-16 20:41:08.739564] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 25, size: 327680000, lookups: 125, failures: 0, skipped: 7 >[2014-06-16 20:41:08.740108] W [glusterfsd.c:1182:cleanup_and_exit] (--> 0-: received signum (15), shutting down >[2014-06-16 20:48:05.729605] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.17 (args: /usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=e1ecdfdd-114d-40af-8b02-78fe44595c47 --socket-file /var/run/gluster/gluster-rebalance-c38f85bc-c244-4e5d-a101-7e0a9965f909.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/e1ecdfdd-114d-40af-8b02-78fe44595c47.pid -l /var/log/glusterfs/hosdu-rebalance.log) >[2014-06-16 20:48:10.727378] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value 'e1ecdfdd-114d-40af-8b02-78fe44595c47' >[2014-06-16 20:48:10.727406] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1' >[2014-06-16 20:48:10.727419] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on' >[2014-06-16 20:48:10.727428] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:48:10.727435] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:48:10.727443] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes' >[2014-06-16 20:48:10.727450] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:48:10.727460] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:48:10.727471] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:48:10.727478] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off' >[2014-06-16 20:48:10.727487] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:48:10.727494] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:48:10.727500] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:48:10.727507] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off' >[2014-06-16 20:48:10.727515] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:48:10.727522] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:48:10.727529] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:48:10.727536] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off' >[2014-06-16 20:48:10.727543] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:48:10.727550] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:48:10.727557] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:48:10.727563] I [graph.c:269:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off' >[2014-06-16 20:48:10.728102] I [dht-shared.c:332:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-06-16 20:48:10.734090] I [client.c:2280:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.738410] I [client.c:2280:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.741613] I [client.c:2280:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.744618] I [client.c:2280:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.747801] I [client.c:2280:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.750806] I [client.c:2280:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.754126] I [client.c:2280:notify] 0-hosdu-client-8: parent translators are ready, attempting connect on transport >[2014-06-16 20:48:10.757232] I [client.c:2280:notify] 0-hosdu-client-9: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume hosdu-client-2 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host rhsauto056.lab.eng.blr.redhat.com > 5: option remote-subvolume /bricks/hosdu_brick2 > 6: option transport-type socket > 7: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 8: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 9: option send-gids true > 10: end-volume > 11: > 12: volume hosdu-client-3 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host rhsauto057.lab.eng.blr.redhat.com > 16: option remote-subvolume /bricks/hosdu_brick3 > 17: option transport-type socket > 18: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 19: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 20: option send-gids true > 21: end-volume > 22: > 23: volume hosdu-replicate-0 > 24: type cluster/replicate > 25: option data-self-heal off > 26: option metadata-self-heal off > 27: option entry-self-heal off > 28: option readdir-failover off > 29: subvolumes hosdu-client-2 hosdu-client-3 > 30: end-volume > 31: > 32: volume hosdu-client-4 > 33: type protocol/client > 34: option ping-timeout 42 > 35: option remote-host rhsauto056.lab.eng.blr.redhat.com > 36: option remote-subvolume /bricks/hosdu_brick4 > 37: option transport-type socket > 38: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 39: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 40: option send-gids true > 41: end-volume > 42: > 43: volume hosdu-client-5 > 44: type protocol/client > 45: option ping-timeout 42 > 46: option remote-host rhsauto057.lab.eng.blr.redhat.com > 47: option remote-subvolume /bricks/hosdu_brick5 > 48: option transport-type socket > 49: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 50: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 51: option send-gids true > 52: end-volume > 53: > 54: volume hosdu-replicate-1 > 55: type cluster/replicate > 56: option data-self-heal off > 57: option metadata-self-heal off > 58: option entry-self-heal off > 59: option readdir-failover off > 60: subvolumes hosdu-client-4 hosdu-client-5 > 61: end-volume > 62: > 63: volume hosdu-client-6 > 64: type protocol/client > 65: option ping-timeout 42 > 66: option remote-host rhsauto022.lab.eng.blr.redhat.com > 67: option remote-subvolume /bricks/hosdu_brick6 > 68: option transport-type socket > 69: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 70: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 71: option send-gids true > 72: end-volume > 73: > 74: volume hosdu-client-7 > 75: type protocol/client > 76: option ping-timeout 42 > 77: option remote-host rhsauto022.lab.eng.blr.redhat.com > 78: option remote-subvolume /bricks/hosdu_brick8 > 79: option transport-type socket > 80: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 > 81: option password d467cbcc-4275-45fb-988e-7bcd55923c21 > 82: option send-gids true > 83: end-volume > 84: > 85: volume hosdu-replicate-2 > 86: type cluster/replicate > 87: option data-self-heal off > 88: option metadata-self-heal off > 89: option entry-self-heal off > 90: option readdir-failover off > 91: subvolumes hosdu-client-6 hosdu-client-7 > 92: end-volume > 93: > 94: volume hosdu-client-8 > 95: type protocol/client > 96: option ping-timeout 42 > 97: option remote-host rhsauto022.lab.eng.blr.redhat.com > 98: option remote-subvolume /bricks/hosdu_brick9 > 99: option transport-type socket >100: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 >101: option password d467cbcc-4275-45fb-988e-7bcd55923c21 >102: option send-gids true >103: end-volume >104: >105: volume hosdu-client-9 >106: type protocol/client >107: option ping-timeout 42 >108: option remote-host rhsauto022.lab.eng.blr.redhat.com >109: option remote-subvolume /bricks/hosdu_brick10 >110: option transport-type socket >111: option username e61519f5-94f4-4e5b-b0ee-b55bebe09060 >112: option password d467cbcc-4275-45fb-988e-7bcd55923c21 >113: option send-gids true >114: end-volume >115: >116: volume hosdu-replicate-3 >117: type cluster/replicate >118: option data-self-heal off >119: option metadata-self-heal off >120: option entry-self-heal off >121: option readdir-failover off >122: subvolumes hosdu-client-8 hosdu-client-9 >123: end-volume >124: >125: volume hosdu-dht >126: type cluster/distribute >127: option use-readdirp yes >128: option lookup-unhashed yes >129: option assert-no-child-down yes >130: option readdir-optimize on >131: option rebalance-cmd 1 >132: option node-uuid e1ecdfdd-114d-40af-8b02-78fe44595c47 >133: subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3 >134: end-volume >135: >136: volume hosdu-write-behind >137: type performance/write-behind >138: subvolumes hosdu-dht >139: end-volume >140: >141: volume hosdu-read-ahead >142: type performance/read-ahead >143: subvolumes hosdu-write-behind >144: end-volume >145: >146: volume hosdu-readdir-ahead >147: type performance/readdir-ahead >148: subvolumes hosdu-read-ahead >149: end-volume >150: >151: volume hosdu-io-cache >152: type performance/io-cache >153: subvolumes hosdu-readdir-ahead >154: end-volume >155: >156: volume hosdu-quick-read >157: type performance/quick-read >158: subvolumes hosdu-io-cache >159: end-volume >160: >161: volume hosdu-md-cache >162: type performance/md-cache >163: subvolumes hosdu-quick-read >164: end-volume >165: >166: volume hosdu >167: type debug/io-stats >168: option latency-measurement on >169: option count-fop-hits on >170: subvolumes hosdu-md-cache >171: end-volume >172: >+------------------------------------------------------------------------------+ >[2014-06-16 20:48:10.762241] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0) >[2014-06-16 20:48:10.765694] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0) >[2014-06-16 20:48:10.769198] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:10.769432] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-4: Connected to hosdu-client-4, attached to remote volume '/bricks/hosdu_brick4'. >[2014-06-16 20:48:10.769446] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:10.769569] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-4' came back up; going online. >[2014-06-16 20:48:10.769610] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:10.769672] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1 >[2014-06-16 20:48:10.770135] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-2: Connected to hosdu-client-2, attached to remote volume '/bricks/hosdu_brick2'. >[2014-06-16 20:48:10.770149] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:10.770206] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-2' came back up; going online. >[2014-06-16 20:48:10.770363] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1 >[2014-06-16 20:48:15.743862] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0) >[2014-06-16 20:48:15.748023] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49155 (from 0) >[2014-06-16 20:48:15.748166] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-8: changing port to 49156 (from 0) >[2014-06-16 20:48:15.748217] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-9: changing port to 49157 (from 0) >[2014-06-16 20:48:15.770315] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0) >[2014-06-16 20:48:15.778869] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0) >[2014-06-16 20:48:15.780289] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.788192] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.788425] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.788892] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-6: Connected to hosdu-client-6, attached to remote volume '/bricks/hosdu_brick6'. >[2014-06-16 20:48:15.788926] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.789745] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-6' came back up; going online. >[2014-06-16 20:48:15.789814] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.790371] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1 >[2014-06-16 20:48:15.790429] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-7: Connected to hosdu-client-7, attached to remote volume '/bricks/hosdu_brick8'. >[2014-06-16 20:48:15.790449] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.791652] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.792135] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-8: Connected to hosdu-client-8, attached to remote volume '/bricks/hosdu_brick9'. >[2014-06-16 20:48:15.792166] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.792580] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-8' came back up; going online. >[2014-06-16 20:48:15.792860] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1 >[2014-06-16 20:48:15.792912] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-3: Connected to hosdu-client-3, attached to remote volume '/bricks/hosdu_brick3'. >[2014-06-16 20:48:15.792931] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.793766] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-9: Connected to hosdu-client-9, attached to remote volume '/bricks/hosdu_brick10'. >[2014-06-16 20:48:15.793937] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.794890] I [client-handshake.c:1415:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-06-16 20:48:15.795296] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1 >[2014-06-16 20:48:15.795489] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-9: Server lk version = 1 >[2014-06-16 20:48:15.795534] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-8: Server lk version = 1 >[2014-06-16 20:48:15.795960] I [client-handshake.c:1200:client_setvolume_cbk] 0-hosdu-client-5: Connected to hosdu-client-5, attached to remote volume '/bricks/hosdu_brick5'. >[2014-06-16 20:48:15.795989] I [client-handshake.c:1212:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-06-16 20:48:15.797789] I [client-handshake.c:188:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1 >[2014-06-16 20:48:15.814794] I [dht-common.c:2601:dht_setxattr] 0-hosdu-dht: fixing the layout of / >[2014-06-16 20:48:15.828918] I [dht-rebalance.c:1220:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on / >[2014-06-16 20:48:15.852763] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /4: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:15.987891] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs >[2014-06-16 20:48:15.996384] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0 >[2014-06-16 20:48:18.217334] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /4 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:18.238902] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /29: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:19.926325] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /29 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:19.932097] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /30: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:21.379081] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /30 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:21.422356] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /74: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:21.549950] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 6.00 secs >[2014-06-16 20:48:21.549993] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 3, size: 39321600, lookups: 26, failures: 0, skipped: 0 >[2014-06-16 20:48:23.630311] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /74 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:23.640393] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /75: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:25.710154] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /75 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:25.720831] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /76: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:26.889392] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 11.00 secs >[2014-06-16 20:48:26.889449] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 5, size: 65536000, lookups: 28, failures: 0, skipped: 0 >[2014-06-16 20:48:27.579995] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /76 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:27.598599] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /90: attempting to move from hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:29.269825] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /90 from subvolume hosdu-replicate-0 to hosdu-replicate-2 >[2014-06-16 20:48:29.299375] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /11: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:31.195233] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /11 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:31.208639] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /20: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:32.211654] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 17.00 secs >[2014-06-16 20:48:32.211701] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 8, size: 104857600, lookups: 42, failures: 0, skipped: 0 >[2014-06-16 20:48:33.257841] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /20 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:33.276720] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /35: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:35.454971] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /35 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:35.461837] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /46: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:37.327490] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /46 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:37.340118] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /49: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:37.562564] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 22.00 secs >[2014-06-16 20:48:37.562657] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 11, size: 144179200, lookups: 50, failures: 0, skipped: 0 >[2014-06-16 20:48:39.043848] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /49 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:39.067002] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /68: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:40.604011] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /68 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:40.619525] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /79: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:42.627333] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /79 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:42.637902] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /85: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:42.884400] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 27.00 secs >[2014-06-16 20:48:42.884447] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 14, size: 183500800, lookups: 64, failures: 0, skipped: 0 >[2014-06-16 20:48:44.668533] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /85 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:44.679070] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /88: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:46.733815] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /88 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:46.743083] I [dht-rebalance.c:764:dht_migrate_file] 0-hosdu-dht: /93: attempting to move from hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:48.197683] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 33.00 secs >[2014-06-16 20:48:48.197760] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 16, size: 209715200, lookups: 67, failures: 0, skipped: 0 >[2014-06-16 20:48:48.617216] I [dht-rebalance.c:985:dht_migrate_file] 0-hosdu-dht: completed migration of /93 from subvolume hosdu-replicate-1 to hosdu-replicate-3 >[2014-06-16 20:48:48.812273] I [dht-rebalance.c:1443:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 32.98 secs >[2014-06-16 20:48:48.840375] I [dht-rebalance.c:1876:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 33.00 secs >[2014-06-16 20:48:48.840430] I [dht-rebalance.c:1879:gf_defrag_status_get] 0-glusterfs: Files migrated: 17, size: 222822400, lookups: 117, failures: 0, skipped: 0 >[2014-06-16 20:48:48.841593] W [glusterfsd.c:1182:cleanup_and_exit] (--> 0-: received signum (15), shutting down
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1110311
: 909580 |
909582
|
911310
|
913381