Attachment 841882 Details for Bug 1046603 – Rebalance logs from node

Rebalance logs from node

hosdu-rebalance.log.node (text/plain), 47.77 KB, created by Lalatendu Mohanty on 2013-12-26 12:12:55 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lalatendu Mohanty

Created: 2013-12-26 12:12:55 UTC

Size: 47.77 KB

patch

obsolete

>[2013-12-25 07:13:15.872845] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb --socket-file /var/lib/glusterd/vols/hosdu/rebalance/6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 07:13:15.874935] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 07:13:15.879608] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 07:13:15.879667] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 07:13:15.880149] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 07:13:15.880221] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 07:13:20.853928] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb'
>[2013-12-25 07:13:20.853953] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 07:13:20.853965] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 07:13:20.853976] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:20.853986] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:20.853996] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:13:20.854006] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:20.854017] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:20.854027] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:20.854037] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:13:20.854048] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:20.854059] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:20.854069] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:20.854078] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:13:20.854088] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:20.854098] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:20.854107] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:20.854124] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:13:20.854135] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:20.854145] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:20.854155] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:20.854214] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:13:20.854660] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 07:13:20.921734] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 07:13:20.921792] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 07:13:20.927253] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 07:13:20.927302] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 07:13:20.933281] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 07:13:20.933324] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 07:13:20.939150] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 07:13:20.939197] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 07:13:20.944790] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 07:13:20.944813] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 07:13:20.950038] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 07:13:20.950060] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 07:13:20.954640] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 07:13:20.954662] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 07:13:20.959213] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 07:13:20.959234] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 07:13:20.959298] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.963838] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.967607] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.971758] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.984126] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.989825] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.995221] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 07:13:20.999245] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick0
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick1
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick2
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick3
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick4
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick5
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick6
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick7
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 07:13:21.005946] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49152 (from 0)
>[2013-12-25 07:13:21.006014] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49152 (from 0)
>[2013-12-25 07:13:21.006167] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49153 (from 0)
>[2013-12-25 07:13:21.006260] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49153 (from 0)
>[2013-12-25 07:13:21.006370] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49154 (from 0)
>[2013-12-25 07:13:21.031243] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49154 (from 0)
>[2013-12-25 07:13:21.035448] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49152 (from 0)
>[2013-12-25 07:13:21.035540] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49153 (from 0)
>[2013-12-25 07:13:21.045384] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.045562] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.045801] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.045903] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.046166] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.046485] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.046513] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49152, attached to remote volume '/bricks/hosdu_brick0'.
>[2013-12-25 07:13:21.046530] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.046626] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-0' came back up; going online.
>[2013-12-25 07:13:21.046668] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.046682] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49152, attached to remote volume '/bricks/hosdu_brick1'.
>[2013-12-25 07:13:21.046691] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.047238] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.047268] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 07:13:21.047279] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.047496] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-3' came back up; going online.
>[2013-12-25 07:13:21.047536] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.047549] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 07:13:21.047559] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.047771] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-5' came back up; going online.
>[2013-12-25 07:13:21.047903] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 07:13:21.047931] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 07:13:21.047949] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 07:13:21.047967] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 07:13:21.047991] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.048003] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 07:13:21.048012] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.048258] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.048540] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 07:13:21.048930] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.048949] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 07:13:21.048959] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.049216] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.049601] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:13:21.049671] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 07:13:21.050603] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.050625] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 07:13:21.050635] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.050694] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-25 07:13:21.051610] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:13:21.051657] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49153, attached to remote volume '/bricks/hosdu_brick7'.
>[2013-12-25 07:13:21.051671] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:13:21.066181] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 07:13:21.066222] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 07:13:21.105293] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 07:13:21.106461] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 07:13:21.106659] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 07:13:21.108341] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 07:13:21.109019] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-3
>[2013-12-25 07:13:21.109125] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-1
>[2013-12-25 07:13:21.111156] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 07:13:21.111229] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-2: selecting local read_child hosdu-client-5
>[2013-12-25 07:13:21.118596] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 07:13:21.347971] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2013-12-25 07:13:21.348018] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 64, failures: 0, skipped: 0
>[2013-12-25 07:13:21.407935] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /85
>[2013-12-25 07:13:21.408133] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /85
>[2013-12-25 07:13:21.421300] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /88
>[2013-12-25 07:13:21.421380] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /88
>[2013-12-25 07:13:21.492630] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /23
>[2013-12-25 07:13:21.492759] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /23
>[2013-12-25 07:13:21.500352] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /27 on hosdu-replicate-3
>[2013-12-25 07:13:21.520011] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /32 on hosdu-replicate-3
>[2013-12-25 07:13:21.550218] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /44
>[2013-12-25 07:13:21.550330] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /44
>[2013-12-25 07:13:21.565963] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /63 on hosdu-replicate-3
>[2013-12-25 07:13:21.596752] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /82
>[2013-12-25 07:13:21.596990] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /82
>[2013-12-25 07:13:21.607361] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-hosdu-dht: deleting stale linkfile /86 on hosdu-replicate-3
>[2013-12-25 07:13:21.629390] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-7: remote operation failed: File exists. Path: /99
>[2013-12-25 07:13:21.629600] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-6: remote operation failed: File exists. Path: /99
>[2013-12-25 07:13:21.647001] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.53 secs
>[2013-12-25 07:13:21.684587] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2013-12-25 07:13:21.684641] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 101, failures: 0, skipped: 0
>[2013-12-25 07:13:21.687511] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f43bd40194d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f43bda4d851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down
>[2013-12-25 07:19:37.980366] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.52rhs (/usr/sbin/glusterfs -s localhost --volfile-id hosdu --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb --socket-file /var/lib/glusterd/vols/hosdu/rebalance/6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb.sock --pid-file /var/lib/glusterd/vols/hosdu/rebalance/6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb.pid -l /var/log/glusterfs/hosdu-rebalance.log)
>[2013-12-25 07:19:37.982295] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2013-12-25 07:19:37.985629] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2013-12-25 07:19:37.985660] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2013-12-25 07:19:37.985915] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2013-12-25 07:19:37.985938] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2013-12-25 07:19:42.978140] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'node-uuid' for volume 'hosdu-dht' with value '6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb'
>[2013-12-25 07:19:42.978196] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'rebalance-cmd' for volume 'hosdu-dht' with value '1'
>[2013-12-25 07:19:42.978217] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'readdir-optimize' for volume 'hosdu-dht' with value 'on'
>[2013-12-25 07:19:42.978236] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'assert-no-child-down' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:42.978255] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'lookup-unhashed' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:42.978273] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-dht: adding option 'use-readdirp' for volume 'hosdu-dht' with value 'yes'
>[2013-12-25 07:19:42.978309] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'readdir-failover' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:42.978345] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'entry-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:42.978365] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'metadata-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:42.978384] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-3: adding option 'data-self-heal' for volume 'hosdu-replicate-3' with value 'off'
>[2013-12-25 07:19:42.978403] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'readdir-failover' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:42.978436] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'entry-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:42.978455] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'metadata-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:42.978473] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-2: adding option 'data-self-heal' for volume 'hosdu-replicate-2' with value 'off'
>[2013-12-25 07:19:42.978491] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'readdir-failover' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:42.978509] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'entry-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:42.978526] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'metadata-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:42.978544] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-1: adding option 'data-self-heal' for volume 'hosdu-replicate-1' with value 'off'
>[2013-12-25 07:19:42.978562] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'readdir-failover' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:42.978579] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'entry-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:42.978596] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'metadata-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:42.978625] I [graph.c:239:gf_add_cmdline_options] 0-hosdu-replicate-0: adding option 'data-self-heal' for volume 'hosdu-replicate-0' with value 'off'
>[2013-12-25 07:19:42.979349] I [dht-shared.c:311:dht_init_regex] 0-hosdu-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2013-12-25 07:19:42.990151] I [socket.c:3505:socket_init] 0-hosdu-client-7: SSL support is NOT enabled
>[2013-12-25 07:19:42.990178] I [socket.c:3520:socket_init] 0-hosdu-client-7: using system polling thread
>[2013-12-25 07:19:42.991383] I [socket.c:3505:socket_init] 0-hosdu-client-6: SSL support is NOT enabled
>[2013-12-25 07:19:42.991405] I [socket.c:3520:socket_init] 0-hosdu-client-6: using system polling thread
>[2013-12-25 07:19:42.992381] I [socket.c:3505:socket_init] 0-hosdu-client-5: SSL support is NOT enabled
>[2013-12-25 07:19:42.992401] I [socket.c:3520:socket_init] 0-hosdu-client-5: using system polling thread
>[2013-12-25 07:19:42.993442] I [socket.c:3505:socket_init] 0-hosdu-client-4: SSL support is NOT enabled
>[2013-12-25 07:19:42.993461] I [socket.c:3520:socket_init] 0-hosdu-client-4: using system polling thread
>[2013-12-25 07:19:42.994404] I [socket.c:3505:socket_init] 0-hosdu-client-3: SSL support is NOT enabled
>[2013-12-25 07:19:42.994424] I [socket.c:3520:socket_init] 0-hosdu-client-3: using system polling thread
>[2013-12-25 07:19:42.995101] I [socket.c:3505:socket_init] 0-hosdu-client-2: SSL support is NOT enabled
>[2013-12-25 07:19:42.995119] I [socket.c:3520:socket_init] 0-hosdu-client-2: using system polling thread
>[2013-12-25 07:19:42.995852] I [socket.c:3505:socket_init] 0-hosdu-client-1: SSL support is NOT enabled
>[2013-12-25 07:19:42.995870] I [socket.c:3520:socket_init] 0-hosdu-client-1: using system polling thread
>[2013-12-25 07:19:42.997050] I [socket.c:3505:socket_init] 0-hosdu-client-0: SSL support is NOT enabled
>[2013-12-25 07:19:42.997070] I [socket.c:3520:socket_init] 0-hosdu-client-0: using system polling thread
>[2013-12-25 07:19:42.997129] I [client.c:2171:notify] 0-hosdu-client-0: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.003270] I [client.c:2171:notify] 0-hosdu-client-1: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.007304] I [client.c:2171:notify] 0-hosdu-client-2: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.011310] I [client.c:2171:notify] 0-hosdu-client-3: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.015635] I [client.c:2171:notify] 0-hosdu-client-4: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.019578] I [client.c:2171:notify] 0-hosdu-client-5: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.023510] I [client.c:2171:notify] 0-hosdu-client-6: parent translators are ready, attempting connect on transport
>[2013-12-25 07:19:43.027147] I [client.c:2171:notify] 0-hosdu-client-7: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume hosdu-client-0
>  2:     type protocol/client
>  3:     option remote-host rhsauto019.lab.eng.blr.redhat.com
>  4:     option remote-subvolume /bricks/hosdu_brick2
>  5:     option transport-type socket
>  6:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
>  7:     option password d94d42e0-803a-405b-8178-23078d9555b8
>  8: end-volume
>  9: 
> 10: volume hosdu-client-1
> 11:     type protocol/client
> 12:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 13:     option remote-subvolume /bricks/hosdu_brick3
> 14:     option transport-type socket
> 15:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 16:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 17: end-volume
> 18: 
> 19: volume hosdu-replicate-0
> 20:     type cluster/replicate
> 21:     option data-self-heal off
> 22:     option metadata-self-heal off
> 23:     option entry-self-heal off
> 24:     option readdir-failover off
> 25:     subvolumes hosdu-client-0 hosdu-client-1
> 26: end-volume
> 27: 
> 28: volume hosdu-client-2
> 29:     type protocol/client
> 30:     option remote-host rhsauto019.lab.eng.blr.redhat.com
> 31:     option remote-subvolume /bricks/hosdu_brick4
> 32:     option transport-type socket
> 33:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 34:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 35: end-volume
> 36: 
> 37: volume hosdu-client-3
> 38:     type protocol/client
> 39:     option remote-host rhsauto008.lab.eng.blr.redhat.com
> 40:     option remote-subvolume /bricks/hosdu_brick5
> 41:     option transport-type socket
> 42:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 43:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 44: end-volume
> 45: 
> 46: volume hosdu-replicate-1
> 47:     type cluster/replicate
> 48:     option data-self-heal off
> 49:     option metadata-self-heal off
> 50:     option entry-self-heal off
> 51:     option readdir-failover off
> 52:     subvolumes hosdu-client-2 hosdu-client-3
> 53: end-volume
> 54: 
> 55: volume hosdu-client-4
> 56:     type protocol/client
> 57:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 58:     option remote-subvolume /bricks/hosdu_brick6
> 59:     option transport-type socket
> 60:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 61:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 62: end-volume
> 63: 
> 64: volume hosdu-client-5
> 65:     type protocol/client
> 66:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 67:     option remote-subvolume /bricks/hosdu_brick8
> 68:     option transport-type socket
> 69:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 70:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 71: end-volume
> 72: 
> 73: volume hosdu-replicate-2
> 74:     type cluster/replicate
> 75:     option data-self-heal off
> 76:     option metadata-self-heal off
> 77:     option entry-self-heal off
> 78:     option readdir-failover off
> 79:     subvolumes hosdu-client-4 hosdu-client-5
> 80: end-volume
> 81: 
> 82: volume hosdu-client-6
> 83:     type protocol/client
> 84:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 85:     option remote-subvolume /bricks/hosdu_brick9
> 86:     option transport-type socket
> 87:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 88:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 89: end-volume
> 90: 
> 91: volume hosdu-client-7
> 92:     type protocol/client
> 93:     option remote-host rhsauto022.lab.eng.blr.redhat.com
> 94:     option remote-subvolume /bricks/hosdu_brick10
> 95:     option transport-type socket
> 96:     option username 4c71e3df-528c-460b-a81c-0da8a731de82
> 97:     option password d94d42e0-803a-405b-8178-23078d9555b8
> 98: end-volume
> 99: 
>100: volume hosdu-replicate-3
>101:     type cluster/replicate
>102:     option data-self-heal off
>103:     option metadata-self-heal off
>104:     option entry-self-heal off
>105:     option readdir-failover off
>106:     subvolumes hosdu-client-6 hosdu-client-7
>107: end-volume
>108: 
>109: volume hosdu-dht
>110:     type cluster/distribute
>111:     option use-readdirp yes
>112:     option lookup-unhashed yes
>113:     option assert-no-child-down yes
>114:     option readdir-optimize on
>115:     option rebalance-cmd 1
>116:     option node-uuid 6b4bdfec-afdc-4a1f-a264-a2e4d9c235cb
>117:     subvolumes hosdu-replicate-0 hosdu-replicate-1 hosdu-replicate-2 hosdu-replicate-3
>118: end-volume
>119: 
>120: volume hosdu-write-behind
>121:     type performance/write-behind
>122:     subvolumes hosdu-dht
>123: end-volume
>124: 
>125: volume hosdu-read-ahead
>126:     type performance/read-ahead
>127:     subvolumes hosdu-write-behind
>128: end-volume
>129: 
>130: volume hosdu-io-cache
>131:     type performance/io-cache
>132:     subvolumes hosdu-read-ahead
>133: end-volume
>134: 
>135: volume hosdu-quick-read
>136:     type performance/quick-read
>137:     subvolumes hosdu-io-cache
>138: end-volume
>139: 
>140: volume hosdu-md-cache
>141:     type performance/md-cache
>142:     subvolumes hosdu-quick-read
>143: end-volume
>144: 
>145: volume hosdu
>146:     type debug/io-stats
>147:     option latency-measurement on
>148:     option count-fop-hits on
>149:     subvolumes hosdu-md-cache
>150: end-volume
>151: 
>+------------------------------------------------------------------------------+
>[2013-12-25 07:19:43.032784] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-1: changing port to 49153 (from 0)
>[2013-12-25 07:19:43.033023] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-3: changing port to 49154 (from 0)
>[2013-12-25 07:19:43.041407] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-0: changing port to 49153 (from 0)
>[2013-12-25 07:19:43.041466] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-2: changing port to 49154 (from 0)
>[2013-12-25 07:19:43.049897] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.050196] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.050307] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-4: changing port to 49152 (from 0)
>[2013-12-25 07:19:43.050445] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.050464] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-1: Connected to 10.70.36.238:49153, attached to remote volume '/bricks/hosdu_brick3'.
>[2013-12-25 07:19:43.050473] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.050585] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-0: Subvolume 'hosdu-client-1' came back up; going online.
>[2013-12-25 07:19:43.054292] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-6: changing port to 49156 (from 0)
>[2013-12-25 07:19:43.054349] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-1: Server lk version = 1
>[2013-12-25 07:19:43.058155] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.058179] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-3: Connected to 10.70.36.238:49154, attached to remote volume '/bricks/hosdu_brick5'.
>[2013-12-25 07:19:43.058200] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.058410] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-1: Subvolume 'hosdu-client-3' came back up; going online.
>[2013-12-25 07:19:43.058445] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-5: changing port to 49155 (from 0)
>[2013-12-25 07:19:43.058490] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-hosdu-client-7: changing port to 49157 (from 0)
>[2013-12-25 07:19:43.058523] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.058614] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.058711] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-3: Server lk version = 1
>[2013-12-25 07:19:43.066380] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.066408] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-0: Connected to 10.70.36.249:49153, attached to remote volume '/bricks/hosdu_brick2'.
>[2013-12-25 07:19:43.066419] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.066848] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.066892] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-2: Connected to 10.70.36.249:49154, attached to remote volume '/bricks/hosdu_brick4'.
>[2013-12-25 07:19:43.066903] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.067237] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-0: Server lk version = 1
>[2013-12-25 07:19:43.067356] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.067462] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-2: Server lk version = 1
>[2013-12-25 07:19:43.067488] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.067873] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.068473] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.068509] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-6: Connected to 10.70.36.252:49156, attached to remote volume '/bricks/hosdu_brick9'.
>[2013-12-25 07:19:43.068524] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.068586] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-3: Subvolume 'hosdu-client-6' came back up; going online.
>[2013-12-25 07:19:43.068630] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.068645] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-7: Connected to 10.70.36.252:49157, attached to remote volume '/bricks/hosdu_brick10'.
>[2013-12-25 07:19:43.068655] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.068758] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.068775] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-4: Connected to 10.70.36.252:49152, attached to remote volume '/bricks/hosdu_brick6'.
>[2013-12-25 07:19:43.068836] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.068890] I [afr-common.c:3913:afr_notify] 0-hosdu-replicate-2: Subvolume 'hosdu-client-4' came back up; going online.
>[2013-12-25 07:19:43.068920] I [client-handshake.c:1677:select_server_supported_programs] 0-hosdu-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2013-12-25 07:19:43.069318] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-7: Server lk version = 1
>[2013-12-25 07:19:43.069485] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-6: Server lk version = 1
>[2013-12-25 07:19:43.069987] I [client-handshake.c:1460:client_setvolume_cbk] 0-hosdu-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2013-12-25 07:19:43.070005] I [client-handshake.c:1474:client_setvolume_cbk] 0-hosdu-client-5: Connected to 10.70.36.252:49155, attached to remote volume '/bricks/hosdu_brick8'.
>[2013-12-25 07:19:43.070015] I [client-handshake.c:1486:client_setvolume_cbk] 0-hosdu-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2013-12-25 07:19:43.070275] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-4: Server lk version = 1
>[2013-12-25 07:19:43.070908] I [client-handshake.c:450:client_set_lk_version_cbk] 0-hosdu-client-5: Server lk version = 1
>[2013-12-25 07:19:43.075550] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-0: added root inode
>[2013-12-25 07:19:43.076747] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-1: added root inode
>[2013-12-25 07:19:43.076965] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-0: selecting local read_child hosdu-client-1
>[2013-12-25 07:19:43.077154] I [afr-common.c:2290:afr_discovery_cbk] 0-hosdu-replicate-1: selecting local read_child hosdu-client-3
>[2013-12-25 07:19:43.077223] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-3: added root inode
>[2013-12-25 07:19:43.078262] I [afr-common.c:2227:afr_set_root_inode_on_first_lookup] 0-hosdu-replicate-2: added root inode
>[2013-12-25 07:19:43.079498] I [dht-common.c:2646:dht_setxattr] 0-hosdu-dht: fixing the layout of /
>[2013-12-25 07:19:43.084721] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-hosdu-dht: migrate data called on /
>[2013-12-25 07:19:43.142734] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-5: remote operation failed: File exists. Path: /75
>[2013-12-25 07:19:43.142788] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-hosdu-client-4: remote operation failed: File exists. Path: /75
>[2013-12-25 07:19:43.381805] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 0.00 secs
>[2013-12-25 07:19:43.381880] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 77, failures: 0, skipped: 0
>[2013-12-25 07:19:43.524405] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-hosdu-dht: Migration operation on dir / took 0.44 secs
>[2013-12-25 07:19:43.564072] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2013-12-25 07:19:43.564128] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 100, failures: 0, skipped: 0
>[2013-12-25 07:19:43.565381] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7fa1c344894d] (-->/lib64/libpthread.so.0(+0x7851) [0x7fa1c3a94851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052ad]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1046603: 841817 | 841881 | 841882