Attachment 862266 Details for Bug 1064321 – Rebalance logs from the second replica node

Rebalance logs from the second replica node

cinder-rebalance2.log (text/x-log), 483.39 KB, created by shilpa on 2014-02-12 12:28:30 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: shilpa

Created: 2014-02-12 12:28:30 UTC

Size: 483.39 KB

patch

obsolete

>[2014-02-10 09:05:47.710145] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 09:05:47.712095] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 09:05:47.719103] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 09:05:47.719127] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 09:05:47.719306] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 09:05:47.719323] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 09:05:52.700824] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-10 09:05:52.700888] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-10 09:05:52.700901] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 09:05:52.700912] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:52.700922] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:52.700938] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:52.700962] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:52.700992] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:52.701016] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:52.701038] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:52.701061] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:52.701084] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:52.701105] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:52.701127] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:52.701165] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:52.701194] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:52.701235] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:52.701259] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:52.701282] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:52.701306] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:52.701328] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:52.701350] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:52.701384] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:52.701414] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:52.701440] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:52.701463] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:52.701486] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:52.701506] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:52.701527] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:52.701542] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:52.701755] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-3
>[2014-02-10 09:05:52.701884] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 09:05:52.751206] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-10 09:05:52.751242] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-10 09:05:52.758176] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-10 09:05:52.758204] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-10 09:05:52.766783] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 09:05:52.766808] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 09:05:52.776350] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 09:05:52.776381] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 09:05:52.782460] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 09:05:52.782489] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 09:05:52.786937] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 09:05:52.786962] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 09:05:52.792099] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 09:05:52.792123] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 09:05:52.797231] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 09:05:52.797261] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 09:05:52.801396] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 09:05:52.801418] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 09:05:52.809653] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 09:05:52.809691] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 09:05:52.819765] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 09:05:52.819793] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 09:05:52.825903] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 09:05:52.825937] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 09:05:52.826048] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.830720] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.835403] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.839497] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.843446] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.847130] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.851341] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.854660] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.858572] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.862102] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.865132] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:52.868192] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.121
>  4:     option remote-subvolume /rhs/brick1/c1
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.140
> 14:     option remote-subvolume /rhs/brick1/c2
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.96
> 34:     option remote-subvolume /rhs/brick1/c3
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.77
> 44:     option remote-subvolume /rhs/brick1/c4
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c5
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c6
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c7
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c8
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c9
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c10
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c11
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c12
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>189:     option decommissioned-bricks cinder-replicate-3
>190:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>191: end-volume
>192: 
>193: volume cinder-write-behind
>194:     type performance/write-behind
>195:     subvolumes cinder-dht
>196: end-volume
>197: 
>198: volume cinder
>199:     type debug/io-stats
>200:     option latency-measurement off
>201:     option count-fop-hits off
>202:     subvolumes cinder-write-behind
>203: end-volume
>204: 
>+------------------------------------------------------------------------------+
>[2014-02-10 09:05:52.873177] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49152 (from 0)
>[2014-02-10 09:05:52.873386] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49153 (from 0)
>[2014-02-10 09:05:52.882005] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49152 (from 0)
>[2014-02-10 09:05:52.882132] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49154 (from 0)
>[2014-02-10 09:05:52.882214] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49153 (from 0)
>[2014-02-10 09:05:52.882292] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 09:05:52.903797] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49153 (from 0)
>[2014-02-10 09:05:52.903900] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49154 (from 0)
>[2014-02-10 09:05:52.903970] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49154 (from 0)
>[2014-02-10 09:05:52.921977] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.922107] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 09:05:52.922174] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49153 (from 0)
>[2014-02-10 09:05:52.922220] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.922316] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49154 (from 0)
>[2014-02-10 09:05:52.922381] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.922680] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.922708] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 09:05:52.922775] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.922906] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 09:05:52.939326] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.939461] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.939557] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.939661] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 09:05:52.939705] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.939787] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.939810] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 09:05:52.939827] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.940205] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.940301] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.940401] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.940425] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49153, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-10 09:05:52.940442] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.940540] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 09:05:52.940740] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.940766] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 09:05:52.940784] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.940925] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-10 09:05:52.940969] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 09:05:52.941074] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.941099] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49153, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-10 09:05:52.941116] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.941379] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.941406] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 09:05:52.941465] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.941555] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-10 09:05:52.941633] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 09:05:52.941684] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.941798] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.941822] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.121:49152, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 09:05:52.941883] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.941983] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 09:05:52.942023] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 09:05:52.942069] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.942166] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.942189] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 09:05:52.942205] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.942295] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-10 09:05:52.942341] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.942360] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 09:05:52.942377] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.942653] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:52.942938] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.942967] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 09:05:52.942984] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.943272] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 09:05:52.943333] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.943354] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 09:05:52.943371] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.943698] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 09:05:52.943738] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-10 09:05:52.943779] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 09:05:52.943809] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-10 09:05:52.943884] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 09:05:52.944033] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:52.944068] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.140:49152, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 09:05:52.944088] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:52.944653] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 09:05:52.944767] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 09:05:53.000438] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 09:05:53.003867] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 09:05:53.003968] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 09:05:53.004051] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 09:05:53.004290] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 09:05:53.004431] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-5: added root inode
>[2014-02-10 09:05:53.005714] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-1: selecting local read_child cinder-client-3
>[2014-02-10 09:05:53.005754] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-10 09:05:53.005779] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-5: selecting local read_child cinder-client-11
>[2014-02-10 09:05:53.007796] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 09:05:53.018051] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 09:05:53.115458] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.10 secs
>[2014-02-10 09:05:53.122876] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2014-02-10 09:05:53.122938] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 09:05:53.123651] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 10:45:51.781438] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 10:45:51.783084] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 10:45:51.785885] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 10:45:51.785909] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 10:45:51.786115] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 10:45:51.786133] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 10:45:56.793406] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-10 10:45:56.798279] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-10 10:45:56.798300] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 10:45:56.798312] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:56.798322] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:56.798331] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:56.798341] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:56.798352] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:56.798361] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:56.798371] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:56.798381] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:56.798390] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:56.798400] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:56.798409] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:56.798419] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:56.798428] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:56.798438] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:56.798447] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:56.798457] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:56.798466] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:56.798484] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:56.798495] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:56.798505] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:56.798514] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:56.798523] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:56.798532] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:56.798733] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 10:45:56.815450] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 10:45:56.815495] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 10:45:56.816176] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 10:45:56.816194] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 10:45:56.817447] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 10:45:56.817467] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 10:45:56.826117] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 10:45:56.826150] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 10:45:56.845287] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 10:45:56.845331] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 10:45:56.858063] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 10:45:56.858089] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 10:45:56.874912] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 10:45:56.874943] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 10:45:56.880224] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 10:45:56.880249] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 10:45:56.885530] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 10:45:56.885556] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 10:45:56.890924] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 10:45:56.890948] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 10:45:56.891017] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.894551] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.897779] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.901676] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.905930] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.909244] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.912393] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.915473] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.918751] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:56.922008] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 4
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 10:45:56.927025] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 10:45:56.927106] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 10:45:56.927171] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 10:45:56.927274] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 10:45:56.940533] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 10:45:56.940610] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 10:45:56.940683] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 10:45:56.940750] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 10:45:56.953536] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 10:45:56.953612] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 10:45:56.960621] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.960775] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.960902] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.960966] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961130] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.961151] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 10:45:56.961161] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.961279] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-10 10:45:56.961421] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961525] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961642] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961709] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961817] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 10:45:56.961852] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.961956] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.961975] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 10:45:56.961985] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.962043] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-10 10:45:56.962076] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.962089] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 10:45:56.962099] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.962201] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-10 10:45:56.962233] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:56.962296] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.962312] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 10:45:56.962322] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.962664] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.962757] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 10:45:56.962794] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.962873] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-10 10:45:56.962917] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.962959] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 10:45:56.962976] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.963058] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 10:45:56.963088] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 10:45:56.963108] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 10:45:56.963158] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.963173] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 10:45:56.963183] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.963338] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 10:45:56.963407] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.963448] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 10:45:56.963461] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.963750] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.963772] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 10:45:56.963782] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.963995] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:56.964015] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 10:45:56.964025] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:56.965368] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 10:45:56.965413] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 10:45:56.965453] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 10:45:56.965477] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 10:45:56.965496] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 10:45:56.965554] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 10:45:56.997332] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 10:45:56.999612] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 10:45:56.999722] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 10:45:56.999812] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 10:45:56.999918] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 10:45:57.000660] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 10:45:57.001300] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-10 10:45:57.001702] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-10 10:45:57.015387] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2014-02-10 10:45:57.015409] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-10 10:45:57.016036] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 10:46:08.060827] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 10:46:08.062265] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 10:46:08.066745] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 10:46:08.066787] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 10:46:08.067097] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 10:46:08.067130] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 10:46:13.061873] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-10 10:46:13.061898] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 10:46:13.061909] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 10:46:13.061920] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:13.061931] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:13.061941] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:13.061958] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:13.061970] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:13.061980] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:13.061989] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:13.062000] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:13.062009] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:13.062019] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:13.062029] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:13.062039] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:13.062048] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:13.062069] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:13.062080] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:13.062090] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:13.062100] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:13.062109] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:13.062119] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:13.062129] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:13.062138] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:13.062148] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:13.062158] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:13.062332] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 10:46:13.069770] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 10:46:13.069810] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 10:46:13.070505] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 10:46:13.070522] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 10:46:13.071210] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 10:46:13.071240] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 10:46:13.071945] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 10:46:13.071962] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 10:46:13.072624] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 10:46:13.072640] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 10:46:13.073346] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 10:46:13.073364] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 10:46:13.074104] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 10:46:13.074122] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 10:46:13.074803] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 10:46:13.074820] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 10:46:13.078992] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 10:46:13.079015] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 10:46:13.083653] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 10:46:13.083678] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 10:46:13.083748] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.087015] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.090204] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.093277] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.096342] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.099391] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.102504] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.105605] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.108789] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:13.112127] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 10:46:13.117099] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 10:46:13.120376] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 10:46:13.120471] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 10:46:13.126793] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 10:46:13.126867] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 10:46:13.133189] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 10:46:13.133266] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 10:46:13.139635] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 10:46:13.139690] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.139795] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 10:46:13.139871] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 10:46:13.149332] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.149410] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.149588] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.149608] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 10:46:13.149618] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.149681] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-10 10:46:13.149716] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.149776] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.149954] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 10:46:13.149982] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.150079] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.150108] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 10:46:13.150119] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.150293] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.150311] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 10:46:13.150321] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.150373] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-10 10:46:13.150401] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.150462] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.150479] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 10:46:13.150489] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.150544] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 10:46:13.150572] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.150639] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.150656] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 10:46:13.150666] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.150799] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.150817] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 10:46:13.150852] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.150925] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-10 10:46:13.150955] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.151011] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 10:46:13.151036] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:13.151112] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 10:46:13.151148] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.151161] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 10:46:13.151171] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.151230] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-10 10:46:13.151256] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 10:46:13.151291] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 10:46:13.151316] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.151328] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 10:46:13.151338] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.151518] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 10:46:13.151550] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.151562] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 10:46:13.151572] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.151739] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:13.151759] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 10:46:13.151768] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:13.152022] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 10:46:13.152050] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 10:46:13.152292] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 10:46:13.152546] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 10:46:13.174331] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 10:46:13.175720] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 10:46:13.175802] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 10:46:13.175966] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 10:46:13.176206] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 10:46:13.176892] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 10:46:13.177676] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-10 10:46:13.177719] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-10 10:46:13.181230] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 10:46:13.206042] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.02 secs
>[2014-02-10 10:46:13.211368] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 10:46:13.211395] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0, skipped: 0
>[2014-02-10 10:46:13.212072] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 11:50:55.626719] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 11:50:55.628704] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 11:50:55.633731] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 11:50:55.633776] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 11:50:55.634135] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 11:50:55.634171] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 11:51:00.623054] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-10 11:51:00.623079] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 11:51:00.623091] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 11:51:00.623103] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:00.623113] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:00.623123] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:00.623134] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:00.623154] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:00.623164] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:00.623174] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:00.623185] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:00.623215] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:00.623228] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:00.623238] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:00.623249] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:00.623267] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:00.623278] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:00.623288] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:00.623298] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:00.623308] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:00.623321] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:00.623332] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:00.623342] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:00.623359] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:00.623376] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:00.623393] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:00.623604] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 11:51:00.795200] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 11:51:00.795260] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 11:51:00.799548] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 11:51:00.799570] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 11:51:00.804235] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 11:51:00.804260] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 11:51:00.808309] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 11:51:00.808333] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 11:51:00.812470] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 11:51:00.812493] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 11:51:00.816627] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 11:51:00.816650] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 11:51:00.821083] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 11:51:00.821107] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 11:51:00.825496] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 11:51:00.825518] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 11:51:00.830120] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 11:51:00.830143] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 11:51:00.833528] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 11:51:00.833578] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 11:51:00.833656] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.837000] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.840176] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.843401] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.846576] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.849719] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.852924] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.877323] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.880556] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:00.883657] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 11:51:00.888447] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 11:51:00.888573] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 11:51:00.900004] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 11:51:00.928724] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:00.928896] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:00.929020] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:00.929447] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:00.929467] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 11:51:00.929477] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:00.929610] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 11:51:00.929655] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:00.929668] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 11:51:00.929678] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:00.929750] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 11:51:00.929815] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 11:51:00.929843] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:00.929856] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 11:51:00.929865] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:00.929959] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 11:51:00.930439] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 11:51:00.930488] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 11:51:00.943301] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 11:51:00.948859] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 11:51:00.952450] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:00.952688] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:00.953071] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:00.953105] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 11:51:00.953116] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:00.953201] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 11:51:00.953400] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:00.953419] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 11:51:00.953437] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:00.953541] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 11:51:00.953648] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 11:51:00.953743] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 11:51:01.007988] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 11:51:01.011982] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 11:51:01.012139] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 11:51:01.019061] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 11:51:01.022489] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 11:51:01.026063] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:01.026298] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:01.026401] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:01.026615] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:01.026746] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:01.026766] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 11:51:01.026777] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:01.027134] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:01.027154] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 11:51:01.027164] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:01.027351] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 11:51:01.027422] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:01.027466] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 11:51:01.027511] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:01.027769] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:01.027789] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 11:51:01.027799] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:01.027994] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:01.028170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 11:51:01.028211] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 11:51:01.028267] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 11:51:01.028898] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:01.028921] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 11:51:01.028931] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:01.029245] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 11:51:01.119570] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 11:51:01.120869] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 11:51:01.121112] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 11:51:01.121382] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 11:51:01.121574] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-10 11:51:01.121636] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-10 11:51:01.121677] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 11:51:01.150317] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 11:51:01.156008] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 11:51:01.169330] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-10 11:51:01.176670] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 11:51:01.176728] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 11:51:01.177359] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 12:32:27.516702] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 12:32:27.517938] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 12:32:27.520652] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 12:32:27.520692] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 12:32:27.520897] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 12:32:27.520916] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 12:32:32.515352] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-10 12:32:32.515398] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 12:32:32.515419] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 12:32:32.515438] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:32.515469] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:32.515486] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:32.515503] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:32.515523] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:32.515541] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:32.515558] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:32.515576] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:32.515645] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:32.515664] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:32.515682] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:32.515700] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:32.515717] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:32.515734] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:32.515751] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:32.515769] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:32.515786] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:32.515803] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:32.515819] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:32.515837] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:32.515866] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:32.515901] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:32.515925] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:32.516281] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 12:32:32.557098] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 12:32:32.557150] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 12:32:32.557959] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 12:32:32.557980] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 12:32:32.559350] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 12:32:32.559369] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 12:32:32.560563] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 12:32:32.560606] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 12:32:32.562279] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 12:32:32.562298] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 12:32:32.563828] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 12:32:32.563847] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 12:32:32.565432] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 12:32:32.565452] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 12:32:32.567044] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 12:32:32.567063] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 12:32:32.568642] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 12:32:32.568662] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 12:32:32.569518] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 12:32:32.569538] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 12:32:32.569629] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.573157] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.576443] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.579633] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.582827] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.586003] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.589136] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.592280] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.595508] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:32.598665] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 12:32:32.603441] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 12:32:32.606783] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 12:32:32.606905] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 12:32:32.606950] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 12:32:32.606991] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 12:32:32.619787] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 12:32:32.619869] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 12:32:32.626672] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.626789] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.626857] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.626929] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.627179] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.627272] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.627290] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 12:32:32.627310] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.627400] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-10 12:32:32.627752] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.627772] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 12:32:32.627782] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.627839] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 12:32:32.627868] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 12:32:32.628056] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.628137] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.628154] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 12:32:32.628163] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.628383] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 12:32:32.628496] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.628512] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 12:32:32.628522] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.628731] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:32.628826] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.628845] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 12:32:32.628854] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.628960] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 12:32:32.629029] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.629045] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 12:32:32.629055] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.629121] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 12:32:32.629148] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 12:32:32.629170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 12:32:32.629430] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 12:32:32.629618] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 12:32:32.660637] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:32.660678] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 12:32:32.660689] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:32.660767] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 12:32:32.661359] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 12:32:33.700810] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-10 12:32:33.700962] I [client.c:2103:client_rpc_notify] 0-cinder-client-2: disconnected from 10.70.37.121:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-10 12:32:33.701143] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-10 12:32:33.701181] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-10 12:32:33.701230] I [client.c:2103:client_rpc_notify] 0-cinder-client-8: disconnected from 10.70.37.121:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-10 12:32:33.701310] I [client.c:2103:client_rpc_notify] 0-cinder-client-4: disconnected from 10.70.37.121:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-10 12:32:33.710933] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 12:32:33.712337] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 12:32:33.712435] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-10 12:32:33.712780] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 12:32:33.713055] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 12:32:33.713151] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-10 12:32:33.713200] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 12:32:33.713423] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 12:32:33.720462] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 12:32:33.726166] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-2: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-10 12:32:33.730886] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-4: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-10 12:32:33.731623] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-10 12:32:33.737498] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 12:32:33.737648] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 12:32:33.738274] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 05:57:11.801343] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 05:57:11.802549] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 05:57:11.805835] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 05:57:11.805861] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 05:57:11.806895] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 05:57:11.806918] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 05:57:17.781941] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 05:57:17.782039] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 05:57:17.782117] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 05:57:17.782179] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:17.782240] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:17.782300] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:17.782351] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:17.782412] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:17.782472] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:17.782531] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:17.782585] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:17.782611] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:17.782672] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:17.782733] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:17.782816] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:17.782933] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:17.782994] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:17.783065] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:17.783124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:17.783181] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:17.783240] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:17.783300] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:17.783353] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:17.783411] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:17.783472] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:17.783529] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:17.783863] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 05:57:17.988161] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 05:57:17.988266] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 05:57:18.003329] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 05:57:18.003392] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 05:57:18.010132] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 05:57:18.010198] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 05:57:18.016871] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 05:57:18.016921] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 05:57:18.025166] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 05:57:18.025214] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 05:57:18.030983] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 05:57:18.031083] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 05:57:18.036424] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 05:57:18.036487] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 05:57:18.043589] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 05:57:18.043672] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 05:57:18.051339] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 05:57:18.051431] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 05:57:18.057926] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 05:57:18.057982] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 05:57:18.058185] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.065064] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.071582] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.077672] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.083425] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.089121] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.094778] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.100895] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.105989] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:18.112590] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 05:57:18.119564] E [socket.c:2158:socket_connect_finish] 0-cinder-client-3: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:18.119726] E [socket.c:2158:socket_connect_finish] 0-cinder-client-5: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:18.121122] E [socket.c:2158:socket_connect_finish] 0-cinder-client-9: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:18.122164] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 05:57:18.122279] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 05:57:18.122353] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:18.135752] I [client.c:2103:client_rpc_notify] 0-cinder-client-1: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:18.135832] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 05:57:18.136001] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 05:57:18.136044] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 05:57:18.155064] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-7: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:18.155275] I [client.c:2103:client_rpc_notify] 0-cinder-client-7: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:18.166986] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:18.167261] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:18.167424] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:18.167588] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:18.167704] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:18.190123] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:18.190199] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 05:57:18.190243] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:18.190418] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 05:57:18.190716] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:18.190760] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 05:57:18.190800] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:18.190961] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 05:57:18.191246] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:18.191286] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 05:57:18.191317] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:18.191542] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 05:57:18.191794] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:18.191824] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 05:57:18.191844] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:18.191981] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 05:57:18.192360] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:18.192462] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 05:57:18.192517] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:18.192824] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 05:57:18.193314] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 05:57:18.193453] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 05:57:18.193537] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 05:57:18.193718] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 05:57:18.193801] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 05:57:18.469671] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 05:57:18.480870] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 05:57:18.481112] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 05:57:18.481300] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 05:57:18.481526] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 05:57:18.482334] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 05:57:18.531480] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 05:57:18.696850] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.17 secs
>[2014-02-11 05:57:18.787145] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 05:57:18.787208] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 05:57:18.787872] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 05:58:20.018381] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 05:58:20.020553] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 05:58:20.024430] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 05:58:20.024455] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 05:58:20.024688] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 05:58:20.024706] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 05:58:25.023848] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 05:58:25.023874] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 05:58:25.023885] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 05:58:25.023897] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.023924] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.023943] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.023961] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.023975] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.023985] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.023994] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.024011] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.024021] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.024030] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.024040] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.024050] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.024060] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.024073] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.024082] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.024092] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.024113] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.024124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.024134] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.024144] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.024163] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.024190] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.024203] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.024462] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 05:58:25.092793] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 05:58:25.092864] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 05:58:25.097752] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 05:58:25.097786] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 05:58:25.102999] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 05:58:25.103026] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 05:58:25.107707] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 05:58:25.107733] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 05:58:25.112962] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 05:58:25.113024] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 05:58:25.117753] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 05:58:25.117799] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 05:58:25.122922] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 05:58:25.122986] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 05:58:25.128756] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 05:58:25.128809] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 05:58:25.133900] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 05:58:25.133950] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 05:58:25.139549] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 05:58:25.139616] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 05:58:25.139765] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.145011] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.150641] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.154648] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.157809] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.161363] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.165180] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.168425] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.173037] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.177148] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 05:58:25.182427] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 05:58:25.182542] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 05:58:25.182736] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 05:58:25.195449] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 05:58:25.195533] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 05:58:25.204845] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 05:58:25.204966] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 05:58:25.212603] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.212849] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.213040] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.213324] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.213347] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 05:58:25.213391] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.213509] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-11 05:58:25.213549] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.213677] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.213801] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 05:58:25.213841] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.213976] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.213995] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 05:58:25.214005] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.214443] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.214464] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 05:58:25.214474] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.214563] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 05:58:25.214674] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.214856] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.214890] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 05:58:25.214905] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.214996] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-11 05:58:25.215029] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 05:58:25.215069] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 05:58:25.215142] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.215163] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 05:58:25.215174] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.215566] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.215721] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 05:58:25.215743] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.215819] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 05:58:25.215934] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 05:58:25.215993] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 05:58:25.216099] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.216130] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 05:58:25.216167] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.216267] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 05:58:25.216377] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 05:58:25.216518] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 05:58:26.141957] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:26.142127] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:26.142236] I [client.c:2103:client_rpc_notify] 0-cinder-client-3: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:26.142428] I [client.c:2103:client_rpc_notify] 0-cinder-client-5: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:26.142633] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:26.142763] I [client.c:2103:client_rpc_notify] 0-cinder-client-9: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:26.178540] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 05:58:26.180743] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 05:58:26.181231] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 05:58:26.181753] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 05:58:26.182212] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 05:58:26.183989] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-11 05:58:26.184058] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 05:58:26.185648] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-11 05:58:26.189478] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 05:58:26.200955] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 05:58:26.207016] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 05:58:26.207157] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 05:58:26.208116] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:06:41.019570] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:06:41.020470] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 06:06:41.023392] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:06:41.023416] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:06:41.023566] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:06:41.023583] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:06:46.797030] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 06:06:46.797090] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:06:46.797124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:06:46.797156] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:06:46.797188] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:06:46.797231] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:06:46.797265] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:06:46.797305] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:06:46.797338] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:06:46.797373] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:06:46.797407] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:06:46.797460] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:06:46.797493] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:06:46.797525] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:06:46.797557] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:06:46.797620] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:06:46.797661] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:06:46.797694] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:06:46.797728] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:06:46.797761] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:06:46.797793] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:06:46.797826] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:06:46.797847] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:06:46.797858] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:06:46.797868] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:06:46.797878] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:06:46.798430] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:06:46.912454] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:06:46.912545] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:06:46.914094] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:06:46.914143] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:06:46.915472] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:06:46.915492] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:06:46.916387] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:06:46.916407] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:06:46.917081] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:06:46.917101] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:06:46.917781] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:06:46.917799] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:06:46.918468] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:06:46.918486] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:06:46.919147] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:06:46.919166] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:06:46.919826] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:06:46.919845] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:06:46.920474] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:06:46.920491] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:06:46.920558] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.924947] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.940273] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.943417] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.960828] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.967020] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:46.983713] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:47.019915] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:47.024007] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:06:47.043504] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:06:47.048168] E [socket.c:2158:socket_connect_finish] 0-cinder-client-0: connection to 10.70.37.96:24007 failed (Connection refused)
>[2014-02-11 06:06:47.048723] E [socket.c:2158:socket_connect_finish] 0-cinder-client-6: connection to 10.70.37.96:24007 failed (Connection refused)
>[2014-02-11 06:06:47.049563] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:06:47.049727] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:06:47.072422] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:06:47.082831] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:06:47.082923] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:06:47.110918] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:06:47.127338] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.138729] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.138836] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.138928] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.139031] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.139480] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.139517] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:06:47.139534] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.139753] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 06:06:47.139842] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.139860] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:06:47.139876] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.140041] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 06:06:47.140090] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.140113] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:06:47.140131] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.140471] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.140498] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:06:47.140517] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.140764] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 06:06:47.140823] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.140875] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:06:47.140893] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.141109] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.141282] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:06:47.141319] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:06:47.141348] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:06:47.141379] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:06:47.141407] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:06:47.141850] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.141887] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:06:47.141907] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.152633] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:06:47.288398] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 06:06:47.288470] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 06:06:47.310554] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.310696] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:06:47.311123] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.311157] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 06:06:47.311179] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.311252] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-11 06:06:47.311519] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 06:06:47.321668] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:06:47.321698] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 06:06:47.321715] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:06:47.321792] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-11 06:06:47.322247] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 06:06:47.359671] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 06:06:47.360675] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 06:06:47.360763] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-11 06:06:47.360802] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 06:06:47.360935] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 06:06:47.361380] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 06:06:47.361676] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:06:47.363990] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-11 06:06:47.366160] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:06:47.372686] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-3: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-11 06:06:47.375528] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 06:06:47.395550] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 06:06:47.395579] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:06:47.396278] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:27:31.634546] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:27:31.635760] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 06:27:31.641016] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:27:31.641061] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:27:31.641378] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:27:31.641409] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:27:36.645774] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 06:27:36.645802] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:27:36.645819] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:27:36.645832] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:36.645863] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:36.645874] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:36.645885] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:36.645896] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:36.645907] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:36.645917] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:36.645927] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:36.645937] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:36.645947] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:36.645957] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:36.645967] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:36.645977] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:36.645987] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:36.645996] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:36.646006] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:36.646016] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:36.646026] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:36.646036] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:36.646046] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:36.646056] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:36.646066] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:36.646076] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:36.646278] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:27:36.694539] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:27:36.694655] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:27:36.695389] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:27:36.695410] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:27:36.696124] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:27:36.696143] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:27:36.696802] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:27:36.696831] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:27:36.697480] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:27:36.697498] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:27:36.698194] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:27:36.698214] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:27:36.698879] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:27:36.698898] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:27:36.699538] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:27:36.699556] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:27:36.700211] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:27:36.700230] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:27:36.700879] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:27:36.700898] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:27:36.700961] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.704257] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.707461] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.710544] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.713666] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.716826] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.719929] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.723006] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.726186] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:36.729241] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:27:36.733976] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:27:36.734088] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 06:27:36.734189] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:27:36.743490] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:27:36.743543] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 06:27:36.743631] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:27:36.753065] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:27:36.753119] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:27:36.759750] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.759895] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760020] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760144] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760230] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.760249] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 06:27:36.760320] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.760433] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-11 06:27:36.760473] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760634] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760828] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.760995] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.761016] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:27:36.761026] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.761092] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 06:27:36.761122] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 06:27:36.761145] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:36.761206] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.761266] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:27:36.761280] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.761337] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-11 06:27:36.761426] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.761445] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:27:36.761455] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.761702] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.761722] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:27:36.761787] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.761872] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-11 06:27:36.761907] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:27:36.761949] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:27:36.761982] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:27:36.762070] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.762099] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 06:27:36.762122] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.762336] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-11 06:27:36.762377] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.762391] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:27:36.762409] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.762558] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 06:27:36.762611] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:27:36.762642] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:36.762655] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:27:36.762665] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:36.762948] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:27:36.763226] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:27:37.690293] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:27:37.690429] I [client.c:2103:client_rpc_notify] 0-cinder-client-0: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:27:37.690701] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-6: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:27:37.690801] I [client.c:2103:client_rpc_notify] 0-cinder-client-6: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:27:37.705271] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 06:27:37.705852] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 06:27:37.706314] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 06:27:37.706548] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 06:27:37.706795] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 06:27:37.706911] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-11 06:27:37.707209] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-11 06:27:37.707479] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:27:37.712111] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:27:37.721764] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-3: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-11 06:27:37.725807] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 06:27:37.733358] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 06:27:37.733450] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:27:37.734313] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:31:36.030575] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:31:36.031807] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 06:31:36.036893] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:31:36.036918] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:31:36.037166] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:31:36.037185] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:31:41.161070] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 06:31:41.161096] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:31:41.161107] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:31:41.161118] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:41.161128] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:41.161137] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:41.161147] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:41.161158] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:41.161176] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:41.161185] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:41.161196] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:41.161206] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:41.161215] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:41.161225] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:41.161235] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:41.161244] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:41.161254] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:41.161270] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:41.161281] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:41.161290] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:41.161300] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:41.161309] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:41.161319] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:41.161329] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:41.161338] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:41.161347] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:41.161554] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:31:41.169752] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:31:41.169816] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:31:41.170494] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:31:41.170513] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:31:41.171189] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:31:41.171208] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:31:41.172033] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:31:41.172054] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:31:41.172865] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:31:41.172886] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:31:41.173630] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:31:41.173649] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:31:41.174317] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:31:41.174336] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:31:41.175046] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:31:41.175064] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:31:41.176178] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:31:41.176199] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:31:41.177205] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:31:41.177228] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:31:41.177294] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.180916] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.185503] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.188960] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.192372] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.195638] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.199047] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.202366] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.205503] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:41.208577] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:31:41.213223] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 06:31:41.216515] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 06:31:41.216619] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:31:41.216655] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:31:41.216684] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:31:41.216801] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:31:41.216962] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 06:31:41.236092] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:31:41.236189] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:31:41.236253] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 06:31:41.246286] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246385] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246444] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246512] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246567] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246681] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.246782] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.247027] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.247074] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 06:31:41.247086] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.247153] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 06:31:41.247194] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.247207] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 06:31:41.247218] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.247320] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 06:31:41.247353] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.247367] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:31:41.247427] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.247473] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-11 06:31:41.247503] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 06:31:41.247564] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.247606] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 06:31:41.247648] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.247858] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.247950] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.248018] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:31:41.248033] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.248203] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.248270] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:41.248385] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 06:31:41.248412] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 06:31:41.248438] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.248451] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:31:41.248490] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.248574] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 06:31:41.248637] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.248651] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:31:41.248661] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.248727] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 06:31:41.248779] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:31:41.248852] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.248867] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:31:41.248877] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.249075] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:31:41.249109] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.249145] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 06:31:41.249157] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.249332] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:41.249351] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:31:41.249360] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:41.249785] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:31:41.249825] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 06:31:41.249845] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:31:41.249864] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:31:41.250044] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:31:41.257865] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 06:31:41.259283] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 06:31:41.259710] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 06:31:41.260124] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 06:31:41.260356] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-7
>[2014-02-11 06:31:41.260422] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 06:31:41.260629] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-1
>[2014-02-11 06:31:41.261394] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:31:41.266372] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:31:41.281126] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 06:31:41.290529] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 06:31:41.290608] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:31:41.291536] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:46:07.539743] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:46:07.548060] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:46:07.548151] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:46:07.548801] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:46:07.548829] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:46:13.257869] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-11 06:46:13.257895] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:46:13.257930] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:46:13.257944] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:46:13.257954] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:46:13.257964] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:46:13.257974] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:46:13.257985] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:46:13.257996] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:46:13.258017] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:46:13.258028] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:46:13.258038] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:46:13.258048] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:46:13.258058] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:46:13.258068] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:46:13.258078] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:46:13.258088] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:46:13.258098] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:46:13.258108] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:46:13.258118] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:46:13.258128] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:46:13.258138] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:46:13.258148] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:46:13.258158] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:46:13.258168] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:46:13.258177] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:46:13.258472] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:46:13.746268] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:46:13.746339] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:46:13.779555] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:46:13.779585] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:46:13.793954] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:46:13.793993] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:46:13.810065] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:46:13.810091] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:46:13.822576] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:46:13.822610] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:46:13.824261] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:46:13.824282] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:46:13.838923] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:46:13.838949] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:46:13.883147] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:46:13.883179] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:46:13.898287] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:46:13.898312] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:46:13.915279] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:46:13.915324] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:46:13.927503] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:13.943091] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:13.956861] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:13.977185] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:13.998976] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:14.011801] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:14.032131] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:14.048691] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:14.074586] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:46:14.087432] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:46:14.118596] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 06:46:14.118669] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:46:14.118806] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:46:14.118851] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:46:14.118886] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-7: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:46:14.135578] I [client.c:2103:client_rpc_notify] 0-cinder-client-1: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:46:14.185079] I [client.c:2103:client_rpc_notify] 0-cinder-client-7: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:46:14.185135] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:46:14.185213] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 06:46:14.185312] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:46:14.282485] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:46:14.282569] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:46:14.321674] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322122] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322238] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322392] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322732] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322822] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.322904] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.323207] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.323229] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 06:46:14.323239] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.323303] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 06:46:14.323992] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.324019] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:46:14.324030] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.324094] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-11 06:46:14.324129] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.324143] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:46:14.324153] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.324449] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 06:46:14.324483] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 06:46:14.324529] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:46:14.324605] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.324832] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:46:14.324849] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.324964] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 06:46:14.337557] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.337581] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 06:46:14.337593] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.337651] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 06:46:14.337773] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.337826] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:46:14.337838] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.338084] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.338104] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:46:14.338114] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.338303] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:46:14.338344] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:46:14.338358] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:46:14.338421] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:46:14.339257] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:46:14.339288] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 06:46:14.339307] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:46:14.339324] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:46:14.341542] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:46:14.341594] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:46:14.557553] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:46:14.564835] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:46:14.579012] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 06:46:14.585747] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 06:46:14.585775] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:46:14.586551] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 05:12:35.336805] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 05:12:35.345233] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 05:12:35.345337] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 05:12:35.345783] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 05:12:35.345818] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 05:12:40.340317] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 05:12:40.340344] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 05:12:40.340356] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 05:12:40.340368] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:40.340379] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:40.340389] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:40.340400] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:40.340411] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:40.340423] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:40.340433] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:40.340474] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:40.340488] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:40.340499] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:40.340509] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:40.340519] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:40.340530] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:40.340540] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:40.340550] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:40.340561] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:40.340571] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:40.340581] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:40.340591] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:40.340609] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:40.340696] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:40.340737] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:40.340750] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:40.341345] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-3
>[2014-02-12 05:12:40.341602] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 05:12:40.412882] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 05:12:40.412928] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 05:12:40.416745] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 05:12:40.416766] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 05:12:40.421610] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 05:12:40.421637] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 05:12:40.425198] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 05:12:40.425220] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 05:12:40.428596] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 05:12:40.428617] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 05:12:40.432052] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 05:12:40.432071] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 05:12:40.435467] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 05:12:40.435486] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 05:12:40.438894] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 05:12:40.438922] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 05:12:40.442279] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 05:12:40.442300] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 05:12:40.448066] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 05:12:40.448097] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 05:12:40.448189] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.451573] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.454833] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.458063] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.461369] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.464496] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.467687] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.470851] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.474134] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:40.477209] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 5
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     option decommissioned-bricks cinder-replicate-3
>160:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>161: end-volume
>162: 
>163: volume cinder-write-behind
>164:     type performance/write-behind
>165:     subvolumes cinder-dht
>166: end-volume
>167: 
>168: volume cinder
>169:     type debug/io-stats
>170:     option latency-measurement off
>171:     option count-fop-hits off
>172:     subvolumes cinder-write-behind
>173: end-volume
>174: 
>+------------------------------------------------------------------------------+
>[2014-02-12 05:12:40.481941] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 05:12:40.482065] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 05:12:40.488686] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 05:12:40.488744] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-12 05:12:40.488787] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 05:12:40.488830] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 05:12:40.488861] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 05:12:40.505383] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-12 05:12:40.505449] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 05:12:40.512339] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 05:12:40.516214] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.516325] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.516452] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.516575] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.516831] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.516875] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 05:12:40.516887] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.516988] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 05:12:40.517144] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.517166] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 05:12:40.517177] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.517298] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-12 05:12:40.517330] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.517413] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.517496] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.517615] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 05:12:40.517640] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 05:12:40.517665] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.517734] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 05:12:40.517753] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.517872] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 05:12:40.517938] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.518062] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.518089] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 05:12:40.518101] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.518416] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.518574] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 05:12:40.518609] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.518621] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 05:12:40.518659] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.518760] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 05:12:40.518826] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 05:12:40.518858] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.518907] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 05:12:40.518962] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.519073] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 05:12:40.519144] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.519164] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 05:12:40.519174] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.519507] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.519529] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 05:12:40.519540] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.519742] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:40.519971] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 05:12:40.520007] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 05:12:40.520034] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 05:12:40.520063] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 05:12:40.520089] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.520152] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 05:12:40.520164] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.520508] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:40.520549] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 05:12:40.520560] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:40.525306] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 05:12:40.525351] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 05:12:40.551174] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 05:12:40.555299] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 05:12:40.572424] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.02 secs
>[2014-02-12 05:12:40.580470] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 05:12:40.580535] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 05:12:40.581880] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:40:39.549815] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:40:39.556724] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:40:39.556818] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:40:39.557187] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:40:39.557208] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:40:44.532968] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 06:40:44.532992] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-12 06:40:44.533004] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:40:44.533015] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:44.533026] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:44.533036] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:44.533133] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:44.533140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:44.533141] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:44.533151] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:44.533161] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:44.533171] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:44.533181] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:44.533191] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:44.533201] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:44.533210] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:44.533220] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:44.533230] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:44.533240] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:44.533249] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:44.533259] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:44.533524] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:40:44.556167] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:40:44.556221] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:40:44.561072] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:40:44.561163] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:40:44.565528] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:40:44.565560] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:40:44.569579] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:40:44.569601] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:40:44.573713] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:40:44.573738] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:40:44.578454] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:40:44.578478] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:40:44.584056] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:40:44.584080] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:40:44.588465] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:40:44.588486] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:40:44.592821] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:40:44.592843] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:40:44.597159] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:40:44.597182] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:40:44.597250] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.600544] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.603901] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.607254] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.610449] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.613667] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.616832] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.619991] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.623347] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:44.626534] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 4
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:40:44.631261] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:40:44.634819] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:40:44.634904] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:40:44.635014] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:40:44.635057] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:40:44.635140] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:40:44.635273] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:40:44.635314] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:40:44.641845] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:40:44.651992] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:40:44.665281] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.665714] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666131] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666222] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666438] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.666460] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:40:44.666471] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.666573] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 06:40:44.666634] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666700] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666770] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666861] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.666938] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.666955] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:40:44.666965] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.667054] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 06:40:44.667091] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.667103] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:40:44.667199] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.667453] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:40:44.667489] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.667577] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:40:44.667595] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.667683] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 06:40:44.667735] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.667810] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:44.667915] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:40:44.667943] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:40:44.667984] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:40:44.668012] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.668024] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:40:44.668033] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.668096] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-12 06:40:44.668154] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.668170] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:40:44.668204] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.668319] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-12 06:40:44.668359] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.668400] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:40:44.668410] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.668737] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.668758] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:40:44.668768] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.669059] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:40:44.669162] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:40:44.669210] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:40:44.669230] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:40:44.669256] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.669268] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:40:44.669277] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.669479] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:44.669500] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:40:44.669523] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:44.670029] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:40:44.670080] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:40:44.698981] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:40:44.715301] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:40:44.715424] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-12 06:40:44.716314] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:40:54.067267] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:40:54.070714] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:40:54.070753] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:40:54.070887] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:40:54.070903] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:40:59.067955] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 06:40:59.068957] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 06:40:59.068978] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:40:59.068989] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:59.069008] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:59.069019] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:59.069029] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:59.069040] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:59.069050] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:59.069060] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:59.069070] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:59.069079] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:59.069089] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:59.069098] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:59.069108] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:59.069118] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:59.069127] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:59.069137] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:59.069148] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:59.069157] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:59.069167] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:59.069177] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:59.069186] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:59.069196] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:59.069205] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:59.069215] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:59.069434] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:40:59.075675] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:40:59.075718] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:40:59.076337] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:40:59.076353] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:40:59.077014] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:40:59.077032] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:40:59.077657] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:40:59.077674] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:40:59.078287] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:40:59.078302] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:40:59.078944] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:40:59.078962] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:40:59.079583] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:40:59.079600] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:40:59.080215] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:40:59.080232] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:40:59.080863] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:40:59.080881] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:40:59.081516] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:40:59.081534] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:40:59.081615] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.084760] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.087934] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.091023] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.094124] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.097199] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.100309] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.103421] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.106511] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:59.109624] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:40:59.114080] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:40:59.114132] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:40:59.114172] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:40:59.123607] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:40:59.123661] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:40:59.129899] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:40:59.129951] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:40:59.130036] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:40:59.130076] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:40:59.142818] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.142916] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:40:59.142966] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146417] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146497] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146581] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.146599] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:40:59.146609] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.146669] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:40:59.146704] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146762] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146866] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146933] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.146998] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.147015] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:40:59.147025] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.147084] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:40:59.147165] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.147182] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:40:59.147192] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.147251] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 06:40:59.147280] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.147344] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.147361] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:40:59.147409] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.147570] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:40:59.147598] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:59.147684] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.147702] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:40:59.147711] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.147851] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:40:59.147883] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.147895] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:40:59.147905] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.147954] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 06:40:59.147981] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:40:59.148024] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.148037] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:40:59.148047] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.148099] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 06:40:59.148158] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.148173] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:40:59.148183] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.148297] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:40:59.148340] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:40:59.148421] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.148439] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:40:59.148448] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.148571] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:40:59.148649] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:40:59.148672] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:40:59.148746] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:40:59.148971] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:59.148994] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:40:59.149005] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:59.149745] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:40:59.156343] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:40:59.159612] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 06:40:59.175527] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.02 secs
>[2014-02-12 06:40:59.182934] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:40:59.182959] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 06:40:59.183744] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:51:32.942978] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:51:32.956107] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:51:32.956166] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:51:32.956432] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:51:32.956453] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:51:37.935101] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 06:51:37.935123] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-12 06:51:37.935156] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:51:37.935168] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:37.935179] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:37.935189] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:37.935199] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:37.935210] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:37.935220] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:37.935231] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:37.935241] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:37.935251] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:37.935261] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:37.935271] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:37.935290] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:37.935312] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:37.935322] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:37.935332] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:37.935349] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:37.935360] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:37.935397] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:37.935412] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:37.935423] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:37.935446] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:37.935464] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:37.935474] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:37.935485] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:37.935494] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:37.935504] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:37.935514] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:37.935759] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:51:37.996676] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 06:51:37.996736] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 06:51:38.002647] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 06:51:38.002678] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 06:51:38.007783] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:51:38.007835] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:51:38.013659] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:51:38.013701] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:51:38.019971] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:51:38.020003] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:51:38.026204] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:51:38.026245] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:51:38.031960] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:51:38.031998] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:51:38.037717] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:51:38.037740] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:51:38.042828] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:51:38.042852] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:51:38.047320] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:51:38.047341] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:51:38.054644] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:51:38.054702] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:51:38.061870] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:51:38.061912] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:51:38.062076] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.067406] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.072455] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.076695] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.081489] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.085912] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.090735] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.094321] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.098391] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.102217] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.105407] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:38.108509] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 4
>188:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:51:38.114433] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:51:38.119278] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:51:38.119415] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:51:38.119455] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:51:38.119501] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:51:38.119550] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:51:38.119611] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:51:38.119653] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:51:38.119708] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:51:38.119743] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:51:38.156779] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 06:51:38.156880] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 06:51:38.167314] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.167493] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.167585] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.167795] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.167916] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.168010] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.168147] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.168252] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.168344] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.168495] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.168543] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:51:38.168559] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.168657] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:51:38.168718] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.168745] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:51:38.168756] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.168837] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-12 06:51:38.168893] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.168910] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:51:38.168920] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.169030] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 06:51:38.169087] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.169104] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:51:38.169114] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.169245] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 06:51:38.169300] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.169441] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.169460] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:51:38.169488] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.169698] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.169826] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:51:38.169856] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:51:38.169895] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:51:38.169917] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:51:38.169971] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:51:38.170003] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.170038] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:51:38.170048] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.170232] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:38.170320] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.170357] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:51:38.170397] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.170518] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 06:51:38.170586] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.170620] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:51:38.170632] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.170837] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.170857] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:51:38.170883] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.171165] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:51:38.171219] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.171236] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 06:51:38.171246] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.171323] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 06:51:38.171396] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:51:38.171441] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:51:38.171482] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.171498] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 06:51:38.171508] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.171730] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:38.171758] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:51:38.171788] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:38.172197] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 06:51:38.172242] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 06:51:38.172284] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:51:38.172651] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:51:38.212806] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:51:38.242144] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:51:38.242264] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-12 06:51:38.244392] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:51:51.742555] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:51:51.748766] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:51:51.748822] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:51:51.749050] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:51:51.749075] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:51:56.754345] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 06:51:56.754379] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 06:51:56.754399] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:51:56.754418] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:56.754435] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:56.754452] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:56.754470] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:56.754488] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:56.754520] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:56.754539] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:56.754557] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:56.754575] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:56.754592] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:56.754609] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:56.754628] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:56.754645] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:56.754663] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:56.754681] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:56.754699] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:56.754716] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:56.754733] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:56.754757] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:56.754776] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:56.754793] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:56.754811] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:56.754830] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:56.754849] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:56.754867] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:56.754883] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:56.754901] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:56.755251] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:51:56.786705] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 06:51:56.786760] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 06:51:56.791516] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 06:51:56.791548] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 06:51:56.793472] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:51:56.793509] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:51:56.796579] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:51:56.796611] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:51:56.798325] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:51:56.798356] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:51:56.799275] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:51:56.799304] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:51:56.804110] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:51:56.804182] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:51:56.809248] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:51:56.809277] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:51:56.810014] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:51:56.810043] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:51:56.810803] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:51:56.810826] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:51:56.811558] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:51:56.811579] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:51:56.812322] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:51:56.812342] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:51:56.812419] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.815939] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.819229] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.822431] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.825715] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.828930] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.832408] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.835560] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.838858] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.842160] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.845533] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:56.848831] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 1
>188:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:51:56.854050] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:51:56.857310] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:51:56.860583] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:51:56.860641] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:51:56.860672] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:51:56.860713] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:51:56.860753] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:51:56.860794] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 06:51:56.860833] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:51:56.860980] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:51:56.887739] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:51:56.887861] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 06:51:56.894961] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895088] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895193] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895263] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895340] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895406] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895478] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895539] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895602] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895668] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.895844] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.895864] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:51:56.895875] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.895937] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 06:51:56.895978] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.895992] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:51:56.896002] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.896075] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 06:51:56.896168] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.896193] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:51:56.896206] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.896266] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:51:56.896321] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:51:56.896371] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:51:56.896400] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.896413] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:51:56.896423] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.896490] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:51:56.896522] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.896612] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.896629] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:51:56.896640] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.896815] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.896835] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 06:51:56.896860] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.896930] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 06:51:56.896963] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:56.897175] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:51:56.897214] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.897227] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 06:51:56.897237] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.897379] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.897397] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:51:56.897407] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.897468] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 06:51:56.897502] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.897515] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:51:56.897525] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.897664] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.897689] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:51:56.897700] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.897833] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:51:56.897871] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 06:51:56.897891] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:51:56.897910] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:51:56.897954] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 06:51:56.897982] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.897994] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:51:56.898004] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.898227] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:51:56.898378] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:56.898396] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:51:56.898406] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:56.898606] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:51:56.898643] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:51:56.901610] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:51:56.911452] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:51:56.924284] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 06:51:56.963854] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.04 secs
>[2014-02-12 06:51:56.981687] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:51:56.981739] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 06:51:56.982892] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 10:39:33.509936] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 10:39:33.514079] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 10:39:33.514124] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 10:39:33.514306] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 10:39:33.514325] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 10:39:38.485081] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 10:39:38.485122] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 10:39:38.485143] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 10:39:38.485162] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:38.485181] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:38.485198] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:38.485217] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:38.485237] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:38.485255] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:38.485274] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:38.485293] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:38.485311] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:38.485328] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:38.485346] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:38.485411] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:38.485439] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:38.485458] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:38.485476] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:38.485495] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:38.485512] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:38.485530] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:38.485570] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:38.485591] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:38.485608] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:38.485626] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:38.485643] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:38.485661] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:38.485679] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:38.485696] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:38.485713] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:38.486099] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 10:39:38.568534] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 10:39:38.568609] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 10:39:38.569992] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 10:39:38.590137] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 10:39:38.591412] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 10:39:38.591436] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 10:39:38.592967] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 10:39:38.592988] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 10:39:38.596506] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 10:39:38.596532] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 10:39:38.601193] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 10:39:38.601221] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 10:39:38.605445] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 10:39:38.605470] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 10:39:38.611748] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 10:39:38.611777] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 10:39:38.616135] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 10:39:38.616159] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 10:39:38.620406] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 10:39:38.620440] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 10:39:38.625890] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 10:39:38.625917] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 10:39:38.631458] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 10:39:38.631483] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 10:39:38.631563] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.635535] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.638879] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.642508] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.645765] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.649063] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.652555] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.655743] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.659084] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.662682] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.665968] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:38.669148] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 10:39:38.674126] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 10:39:38.677360] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 10:39:38.680913] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 10:39:38.681144] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 10:39:38.681221] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 10:39:38.681273] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 10:39:38.681397] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 10:39:38.684684] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 10:39:38.684783] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 10:39:38.707043] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.707493] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.707530] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 10:39:38.707553] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.707642] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 10:39:38.707734] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.707812] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 10:39:38.707880] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 10:39:38.707956] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 10:39:38.708194] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 10:39:38.708226] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.718749] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.718850] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.718954] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.719107] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.719261] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.719294] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 10:39:38.719313] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.719451] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 10:39:38.719509] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.719526] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 10:39:38.719536] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.719607] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 10:39:38.719656] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.719795] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.720063] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 10:39:38.720107] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 10:39:38.720153] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.720178] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 10:39:38.720200] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.720287] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 10:39:38.720325] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.720342] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 10:39:38.720356] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.720443] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 10:39:38.720504] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.720531] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 10:39:38.720558] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.720829] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.720850] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 10:39:38.720873] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.721053] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.721086] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 10:39:38.721099] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.721172] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 10:39:38.721303] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 10:39:38.721348] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 10:39:38.721442] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 10:39:38.721485] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 10:39:38.721523] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.721538] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 10:39:38.721548] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.721707] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.721828] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.721910] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:38.722035] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 10:39:38.722092] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 10:39:38.722598] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.722624] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 10:39:38.722636] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.723578] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.723600] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 10:39:38.723610] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.723742] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 10:39:38.723799] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:38.723813] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 10:39:38.723832] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:38.724232] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 10:39:38.724377] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 10:39:38.756814] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 10:39:38.764962] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 10:39:38.795009] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.03 secs
>[2014-02-12 10:39:38.807319] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 10:39:38.807352] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 5, failures: 0, skipped: 0
>[2014-02-12 10:39:38.808327] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 11:08:08.341080] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=a7add118-e164-4f3f-8c41-7dc4b0345451 --socket-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/a7add118-e164-4f3f-8c41-7dc4b0345451.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 11:08:08.344996] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 11:08:08.345056] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 11:08:08.345312] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 11:08:08.345333] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 11:08:13.414586] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value 'a7add118-e164-4f3f-8c41-7dc4b0345451'
>[2014-02-12 11:08:13.414622] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 11:08:13.414646] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 11:08:13.414668] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:13.414689] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:13.414710] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:13.414730] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:13.414761] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:13.414783] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:13.414803] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:13.414843] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:13.414865] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:13.414884] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:13.414905] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:13.414925] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:13.414944] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:13.414963] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:13.414983] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:13.415004] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:13.415042] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:13.415064] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:13.415083] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:13.415103] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:13.415158] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:13.415183] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:13.415204] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:13.415225] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:13.415243] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:13.415262] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:13.415283] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:13.434083] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-5
>[2014-02-12 11:08:13.434305] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 11:08:13.508120] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 11:08:13.508186] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 11:08:13.512175] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 11:08:13.512238] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 11:08:13.516355] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 11:08:13.516421] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 11:08:13.520305] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 11:08:13.520326] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 11:08:13.524611] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 11:08:13.524633] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 11:08:13.528822] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 11:08:13.528844] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 11:08:13.532749] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 11:08:13.532780] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 11:08:13.536797] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 11:08:13.536820] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 11:08:13.540958] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 11:08:13.540980] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 11:08:13.544972] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 11:08:13.544994] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 11:08:13.549023] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 11:08:13.549044] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 11:08:13.553175] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 11:08:13.553195] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 11:08:13.553268] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.556580] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.559912] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.563058] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.566221] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.569302] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.572464] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.575579] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.578824] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.582651] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.585792] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:13.588886] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid a7add118-e164-4f3f-8c41-7dc4b0345451
>189:     option decommissioned-bricks cinder-replicate-5
>190:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>191: end-volume
>192: 
>193: volume cinder-write-behind
>194:     type performance/write-behind
>195:     subvolumes cinder-dht
>196: end-volume
>197: 
>198: volume cinder
>199:     type debug/io-stats
>200:     option latency-measurement off
>201:     option count-fop-hits off
>202:     subvolumes cinder-write-behind
>203: end-volume
>204: 
>+------------------------------------------------------------------------------+
>[2014-02-12 11:08:13.593794] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 11:08:13.597120] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 11:08:13.597207] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 11:08:13.604002] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:13.604132] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:13.604253] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:13.604705] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:13.604763] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 11:08:13.604794] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:13.604898] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 11:08:13.604968] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:13.605000] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 11:08:13.605022] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:13.605102] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 11:08:13.605149] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 11:08:13.605187] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:13.605200] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 11:08:13.605210] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:13.605305] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-11' came back up; going online.
>[2014-02-12 11:08:13.605433] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 11:08:13.605520] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 11:08:20.665003] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 11:08:20.669313] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 11:08:20.669448] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 11:08:20.679960] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:20.680094] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:20.680194] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:20.680630] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:20.680657] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 11:08:20.680668] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:20.680768] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 11:08:20.680820] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:20.680835] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 11:08:20.680844] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:20.680971] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 11:08:20.681032] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:20.681048] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 11:08:20.681057] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:20.681144] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 11:08:20.681162] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 11:08:20.681271] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 11:08:20.681426] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 11:08:21.268554] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 11:08:21.271960] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 11:08:21.275201] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 11:08:21.278993] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:21.279182] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:21.279445] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:21.279467] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 11:08:21.279478] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:21.279834] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 11:08:21.279872] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:21.279885] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 11:08:21.279896] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:21.280219] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:21.280424] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 11:08:21.280681] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:21.280729] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 11:08:21.280740] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:21.280991] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 11:08:24.592565] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 11:08:24.598592] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 11:08:24.598766] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 11:08:24.606142] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:24.606219] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:24.606325] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:24.606515] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:24.606537] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 11:08:24.606547] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:24.606809] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 11:08:24.606867] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:24.606882] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 11:08:24.606892] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:24.607507] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 11:08:24.608166] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:24.608192] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 11:08:24.608203] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:24.608717] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 11:08:24.644076] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 11:08:24.660724] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 11:08:24.713921] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.05 secs
>[2014-02-12 11:08:24.725095] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 11:08:24.725145] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 6, failures: 0, skipped: 0
>[2014-02-12 11:08:24.726433] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3a912e894d] (-->/lib64/libpthread.so.0() [0x3a91607851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1064321: 862265 | 862266