Attachment 862265 Details for Bug 1064321 – Rebalance logs from the replica pair

Rebalance logs from the replica pair

cinder-rebalance.log (text/x-log), 534.31 KB, created by shilpa on 2014-02-12 12:27:53 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: shilpa

Created: 2014-02-12 12:27:53 UTC

Size: 534.31 KB

patch

obsolete

>[2014-02-10 09:05:48.948350] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 09:05:48.949320] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 09:05:48.954486] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 09:05:48.954514] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 09:05:48.954818] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 09:05:48.954868] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 09:05:53.964532] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-10 09:05:53.964555] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-10 09:05:53.964567] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 09:05:53.964618] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:53.964631] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:53.964641] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 09:05:53.964651] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:53.964663] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:53.964672] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:53.964682] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-10 09:05:53.964700] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:53.964710] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:53.964720] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:53.964729] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 09:05:53.964739] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:53.964756] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:53.964775] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:53.964786] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 09:05:53.964862] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:53.964869] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:53.964869] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:53.964869] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 09:05:53.964869] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:53.964871] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:53.964881] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:53.964891] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 09:05:53.964901] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:53.964911] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:53.964921] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:53.964931] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 09:05:53.965343] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-3
>[2014-02-10 09:05:53.965398] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 09:05:53.987318] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-10 09:05:53.987348] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-10 09:05:53.988107] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-10 09:05:53.988129] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-10 09:05:53.988770] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 09:05:53.988788] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 09:05:53.989531] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 09:05:53.989551] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 09:05:53.990849] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 09:05:53.990872] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 09:05:53.994234] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 09:05:53.994255] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 09:05:53.997678] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 09:05:53.997707] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 09:05:53.999148] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 09:05:53.999169] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 09:05:54.002673] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 09:05:54.002726] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 09:05:54.006327] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 09:05:54.006348] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 09:05:54.009963] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 09:05:54.009984] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 09:05:54.013503] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 09:05:54.013524] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 09:05:54.013597] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.016803] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.021159] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.025480] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.029431] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.033009] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.036932] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.040198] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.043483] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.047437] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.050730] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-10 09:05:54.054110] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.121
>  4:     option remote-subvolume /rhs/brick1/c1
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.140
> 14:     option remote-subvolume /rhs/brick1/c2
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.96
> 34:     option remote-subvolume /rhs/brick1/c3
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.77
> 44:     option remote-subvolume /rhs/brick1/c4
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c5
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c6
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c7
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c8
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c9
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c10
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c11
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c12
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     option decommissioned-bricks cinder-replicate-3
>190:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>191: end-volume
>192: 
>193: volume cinder-write-behind
>194:     type performance/write-behind
>195:     subvolumes cinder-dht
>196: end-volume
>197: 
>198: volume cinder
>199:     type debug/io-stats
>200:     option latency-measurement off
>201:     option count-fop-hits off
>202:     subvolumes cinder-write-behind
>203: end-volume
>204: 
>+------------------------------------------------------------------------------+
>[2014-02-10 09:05:54.060057] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49152 (from 0)
>[2014-02-10 09:05:54.060194] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49153 (from 0)
>[2014-02-10 09:05:54.060310] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49154 (from 0)
>[2014-02-10 09:05:54.060354] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49152 (from 0)
>[2014-02-10 09:05:54.060430] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49153 (from 0)
>[2014-02-10 09:05:54.063805] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 09:05:54.080793] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49153 (from 0)
>[2014-02-10 09:05:54.080871] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49154 (from 0)
>[2014-02-10 09:05:54.080949] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49154 (from 0)
>[2014-02-10 09:05:54.091389] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.091471] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.091535] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.091597] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.091666] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.091725] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 09:05:54.091771] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49153 (from 0)
>[2014-02-10 09:05:54.091811] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49154 (from 0)
>[2014-02-10 09:05:54.092243] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.092265] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49153, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-10 09:05:54.092275] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.092343] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-10 09:05:54.092386] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.092399] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 09:05:54.092409] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.092466] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-11' came back up; going online.
>[2014-02-10 09:05:54.092498] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.092511] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 09:05:54.092538] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.092621] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 09:05:54.092656] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.092670] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49153, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-10 09:05:54.092679] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.092911] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.092930] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 09:05:54.092940] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.102520] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.102635] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 09:05:54.102665] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-10 09:05:54.102684] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 09:05:54.102702] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 09:05:54.102719] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 09:05:54.102738] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.102799] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.102912] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.103132] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.103151] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.121:49152, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 09:05:54.103161] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.103219] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 09:05:54.103332] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.103350] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 09:05:54.103359] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.103408] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-10 09:05:54.103438] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.103451] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 09:05:54.103461] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.103624] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.103644] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 09:05:54.103654] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.103711] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-10 09:05:54.103742] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 09:05:54.103761] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-10 09:05:54.103781] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.103858] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 09:05:54.103885] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.103945] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 09:05:54.104054] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 09:05:54.104322] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.104371] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.140:49152, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 09:05:54.104441] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.104754] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.104834] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 09:05:54.104868] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.105181] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 09:05:54.105217] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 09:05:54.105235] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 09:05:54.105763] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 09:05:54.105861] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 09:05:54.105920] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 09:05:54.451692] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 09:05:54.452755] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 09:05:54.453793] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 09:05:54.453905] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 09:05:54.453975] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 09:05:54.454040] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-5: added root inode
>[2014-02-10 09:05:54.454988] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-10 09:05:54.455030] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-5: selecting local read_child cinder-client-10
>[2014-02-10 09:05:54.455253] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 09:05:54.457057] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-1: selecting local read_child cinder-client-2
>[2014-02-10 09:05:54.459713] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 09:05:54.470544] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-15b01eda-b812-4207-be64-ef2320126892: attempting to move from cinder-replicate-3 to cinder-replicate-1
>[2014-02-10 09:08:13.339991] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 139.00 secs
>[2014-02-10 09:08:13.340081] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0, skipped: 0
>[2014-02-10 09:09:27.220504] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-15b01eda-b812-4207-be64-ef2320126892 from subvolume cinder-replicate-3 to cinder-replicate-1
>[2014-02-10 09:09:27.226189] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-00fbcb35-9136-4b16-b3b0-ce37e1e89764: attempting to move from cinder-replicate-3 to cinder-replicate-1
>[2014-02-10 09:09:43.804020] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-00fbcb35-9136-4b16-b3b0-ce37e1e89764 from subvolume cinder-replicate-3 to cinder-replicate-1
>[2014-02-10 09:09:43.808292] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 229.35 secs
>[2014-02-10 09:09:43.818529] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 229.00 secs
>[2014-02-10 09:09:43.818676] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 12884901888, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 09:09:43.820076] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 10:45:52.982732] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 10:45:52.983808] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 10:45:52.986770] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 10:45:52.986796] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 10:45:52.986973] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 10:45:52.986992] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 10:45:57.995932] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-10 10:45:57.995956] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-10 10:45:57.995968] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 10:45:57.995980] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:57.995990] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:57.996000] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:45:57.996010] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:57.996022] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:57.996033] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:57.996043] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:45:57.996054] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:57.996064] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:57.996074] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:57.996084] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:45:57.996104] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:57.996115] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:57.996124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:57.996134] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:45:57.996144] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:57.996154] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:57.996163] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:57.996173] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:45:57.996183] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:57.996192] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:57.996202] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:57.996212] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:45:57.996423] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 10:45:58.025494] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 10:45:58.025533] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 10:45:58.029731] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 10:45:58.029753] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 10:45:58.033946] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 10:45:58.033969] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 10:45:58.038009] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 10:45:58.038030] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 10:45:58.042123] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 10:45:58.042143] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 10:45:58.046199] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 10:45:58.046220] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 10:45:58.050368] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 10:45:58.050390] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 10:45:58.054501] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 10:45:58.054523] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 10:45:58.058592] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 10:45:58.058674] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 10:45:58.063036] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 10:45:58.063059] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 10:45:58.063124] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.066572] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.069666] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.072786] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.075976] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.079130] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.082261] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.085422] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.088533] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 10:45:58.091689] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 4
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 10:45:58.096265] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 10:45:58.096322] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 10:45:58.096390] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 10:45:58.096475] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 10:45:58.103044] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 10:45:58.103105] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 10:45:58.115781] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 10:45:58.119379] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.119487] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.119597] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.119675] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.119743] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.119814] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 10:45:58.119890] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 10:45:58.119939] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.120009] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 10:45:58.120053] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.129590] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.129616] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 10:45:58.129627] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.129697] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 10:45:58.129740] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.129753] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 10:45:58.129764] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.129872] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 10:45:58.129908] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.129922] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 10:45:58.129932] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.129988] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 10:45:58.130020] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.130043] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 10:45:58.130054] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.130116] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 10:45:58.130149] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.130163] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 10:45:58.130172] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.130359] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 10:45:58.130387] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 10:45:58.130406] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 10:45:58.130431] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.130445] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 10:45:58.130454] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.130507] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 10:45:58.130540] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.130553] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 10:45:58.130562] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.130892] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 10:45:58.130922] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 10:45:58.130942] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 10:45:58.130960] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 10:45:58.131120] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.131193] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.131291] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:45:58.131667] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.131690] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 10:45:58.131701] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.132040] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.132070] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 10:45:58.132081] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.132559] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:45:58.132621] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 10:45:58.132635] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:45:58.132971] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 10:45:58.133170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 10:45:58.133620] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 10:45:58.157019] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 10:45:58.158108] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 10:45:58.158278] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 10:45:58.158492] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 10:45:58.158603] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-10 10:45:58.158640] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 10:45:58.158816] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-10 10:45:58.160359] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 10:45:58.172234] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 10:45:58.172294] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-10 10:45:58.173019] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 10:46:09.239997] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 10:46:09.241500] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 10:46:09.244674] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 10:46:09.244696] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 10:46:09.244872] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 10:46:09.244889] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 10:46:14.246059] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-10 10:46:14.246083] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 10:46:14.246094] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 10:46:14.246104] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:14.246114] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:14.246124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 10:46:14.246133] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:14.246144] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:14.246154] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:14.246164] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 10:46:14.246174] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:14.246183] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:14.246193] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:14.246202] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 10:46:14.246212] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:14.246222] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:14.246231] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:14.246241] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 10:46:14.246251] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:14.246260] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:14.246269] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:14.246279] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 10:46:14.246288] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:14.246305] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:14.246315] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:14.246325] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 10:46:14.246525] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 10:46:14.255069] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 10:46:14.255108] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 10:46:14.256197] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 10:46:14.256230] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 10:46:14.257327] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 10:46:14.257361] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 10:46:14.258780] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 10:46:14.258830] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 10:46:14.259942] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 10:46:14.259985] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 10:46:14.261241] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 10:46:14.261284] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 10:46:14.262447] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 10:46:14.262477] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 10:46:14.263673] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 10:46:14.263715] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 10:46:14.268712] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 10:46:14.268750] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 10:46:14.276690] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 10:46:14.276726] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 10:46:14.276836] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.282380] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.288145] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.293855] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.299098] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.304374] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.309463] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.315454] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.320805] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 10:46:14.326154] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 10:46:14.334011] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 10:46:14.334169] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 10:46:14.334240] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 10:46:14.334308] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 10:46:14.334426] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 10:46:14.363873] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 10:46:14.363959] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 10:46:14.376151] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 10:46:14.376243] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 10:46:14.376309] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 10:46:14.394174] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.394333] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.394654] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.394813] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.394925] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.395408] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.395435] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 10:46:14.395452] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.395675] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 10:46:14.395786] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.395809] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 10:46:14.395826] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.395916] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-10 10:46:14.396006] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.396027] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 10:46:14.396043] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.396136] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 10:46:14.396179] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.396291] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.396391] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.396413] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 10:46:14.396429] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.396859] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 10:46:14.396900] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 10:46:14.396929] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 10:46:14.396967] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.396986] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 10:46:14.397003] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.397112] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 10:46:14.397214] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.397264] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 10:46:14.397345] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.397687] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 10:46:14.397730] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.397815] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.397932] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 10:46:14.398062] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.398086] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 10:46:14.398103] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.398431] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 10:46:14.398470] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 10:46:14.398521] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.398573] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 10:46:14.398633] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.398834] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 10:46:14.399081] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.399107] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 10:46:14.399125] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.399385] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 10:46:14.399466] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 10:46:14.399487] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 10:46:14.399503] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 10:46:14.400044] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 10:46:14.400115] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 10:46:14.400233] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 10:46:14.418373] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 10:46:14.419288] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 10:46:14.419558] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 10:46:14.420061] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 10:46:14.420198] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 10:46:14.421356] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 10:46:14.422135] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-10 10:46:14.422182] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-10 10:46:14.432870] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 10:46:14.446483] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-10 10:46:14.452043] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 10:46:14.452075] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 2, failures: 0, skipped: 0
>[2014-02-10 10:46:14.452749] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 11:50:56.881611] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 11:50:56.883936] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 11:50:56.890503] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 11:50:56.890551] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 11:50:56.891046] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 11:50:56.891086] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 11:51:02.142335] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-10 11:51:02.142378] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 11:51:02.142400] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 11:51:02.142421] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:02.142441] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:02.142459] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 11:51:02.142479] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:02.142500] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:02.142536] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:02.142557] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 11:51:02.142577] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:02.142645] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:02.142666] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:02.142685] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 11:51:02.142705] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:02.142779] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:02.142793] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:02.142803] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 11:51:02.142814] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:02.142824] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:02.142839] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:02.142864] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 11:51:02.142886] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:02.142906] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:02.142917] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:02.142927] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 11:51:02.143282] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 11:51:02.228673] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 11:51:02.228715] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 11:51:02.233496] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 11:51:02.233521] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 11:51:02.238839] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 11:51:02.238909] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 11:51:02.244332] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 11:51:02.244356] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 11:51:02.249703] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 11:51:02.249730] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 11:51:02.255427] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 11:51:02.255483] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 11:51:02.261289] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 11:51:02.261345] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 11:51:02.266093] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 11:51:02.266118] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 11:51:02.271331] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 11:51:02.271367] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 11:51:02.279309] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 11:51:02.279342] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 11:51:02.279466] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.285632] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.291463] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.296872] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.302669] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.307739] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.313013] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.318770] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.323972] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 11:51:02.329122] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 11:51:02.336211] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 11:51:02.336358] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 11:51:02.347570] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 11:51:02.347670] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 11:51:02.347732] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 11:51:02.364073] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 11:51:02.370086] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 11:51:02.374728] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.374858] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.374921] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 11:51:02.374964] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 11:51:02.375004] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.375136] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.375332] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.375354] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 11:51:02.375366] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.375430] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-10 11:51:02.375480] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.375494] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 11:51:02.375503] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.375620] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 11:51:02.382378] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.382473] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 11:51:02.382493] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.382612] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 11:51:02.382647] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 11:51:02.382695] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.382844] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 11:51:02.382871] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 11:51:02.382915] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 11:51:02.386141] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.386289] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.386309] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 11:51:02.386319] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.386718] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.386741] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 11:51:02.386800] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.387020] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.387093] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.387113] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 11:51:02.387126] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.387171] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 11:51:02.387253] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 11:51:02.387278] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 11:51:02.387332] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.387417] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.387566] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.387623] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 11:51:02.387638] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.387735] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 11:51:02.387939] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 11:51:02.388104] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 11:51:02.388132] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 11:51:02.388158] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.388170] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 11:51:02.388180] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.388394] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.388435] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 11:51:02.388459] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.388725] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 11:51:02.388755] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 11:51:02.388801] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 11:51:02.388818] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 11:51:02.388851] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 11:51:02.389283] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 11:51:02.423884] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 11:51:02.425261] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 11:51:02.425342] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 11:51:02.425565] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-10 11:51:02.425918] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 11:51:02.426043] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 11:51:02.426177] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-10 11:51:02.426977] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 11:51:02.430895] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 11:51:02.442153] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-10 11:51:02.452390] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 11:51:02.452485] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 11:51:02.453638] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-10 12:32:34.609741] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-10 12:32:34.611386] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-10 12:32:34.616286] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-10 12:32:34.616330] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-10 12:32:34.616630] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-10 12:32:34.616663] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-10 12:32:39.631205] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-10 12:32:39.631232] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-10 12:32:39.631244] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-10 12:32:39.631255] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:39.631265] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:39.631275] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-10 12:32:39.631286] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:39.631297] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:39.631308] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:39.631318] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-10 12:32:39.631328] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:39.631338] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:39.631348] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:39.631456] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-10 12:32:39.631468] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:39.631478] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:39.631499] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:39.631510] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-10 12:32:39.631521] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:39.631530] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:39.631540] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:39.631620] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-10 12:32:39.631640] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:39.631650] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:39.631660] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:39.631697] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-10 12:32:39.631908] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-10 12:32:39.676849] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-10 12:32:39.676907] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-10 12:32:39.677680] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-10 12:32:39.677699] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-10 12:32:39.678789] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-10 12:32:39.678809] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-10 12:32:39.682326] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-10 12:32:39.687968] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-10 12:32:39.694548] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-10 12:32:39.694580] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-10 12:32:39.699818] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-10 12:32:39.699882] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-10 12:32:39.700941] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-10 12:32:39.700969] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-10 12:32:39.701990] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-10 12:32:39.702016] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-10 12:32:39.702943] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-10 12:32:39.702963] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-10 12:32:39.703696] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-10 12:32:39.703715] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-10 12:32:39.703783] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.707396] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.721191] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.733942] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.737008] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.740186] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.743355] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.746545] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.749660] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-10 12:32:39.753082] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-10 12:32:39.757308] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-10 12:32:39.760667] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-10 12:32:39.760929] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-10 12:32:39.764153] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-10 12:32:39.770803] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-10 12:32:39.770884] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-10 12:32:39.770933] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-10 12:32:39.781115] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.781216] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.781286] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.781629] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.781736] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.781755] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-10 12:32:39.781766] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.781872] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-10 12:32:39.781914] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.781987] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.782002] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-10 12:32:39.782015] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.782094] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-10 12:32:39.782128] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.782191] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.782207] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-10 12:32:39.782217] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.782421] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-10 12:32:39.782506] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.782610] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-10 12:32:39.782668] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-10 12:32:39.782889] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.782920] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-10 12:32:39.782940] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.783034] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-10 12:32:39.783134] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.783174] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-10 12:32:39.783190] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.783261] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-10 12:32:39.783297] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.783311] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-10 12:32:39.783321] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.783383] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-10 12:32:39.783410] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-10 12:32:39.783565] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-10 12:32:39.783772] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.783800] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-10 12:32:39.783820] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.784099] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-10 12:32:39.784147] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-10 12:32:39.810090] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-10 12:32:39.816045] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-10 12:32:39.821001] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-10 12:32:39.824623] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.825060] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.825251] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.825282] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-10 12:32:39.825308] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.825565] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-10 12:32:39.825833] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.825912] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-10 12:32:39.825942] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.826267] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-10 12:32:39.826378] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-10 12:32:39.826540] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-10 12:32:39.826569] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-10 12:32:39.826628] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-10 12:32:39.827155] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-10 12:32:39.857371] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-10 12:32:39.859159] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-10 12:32:39.859632] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-10 12:32:39.860160] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-10 12:32:39.860326] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-10 12:32:39.862075] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-10 12:32:39.863182] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-10 12:32:39.863247] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-10 12:32:39.871415] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-10 12:32:39.891656] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.02 secs
>[2014-02-10 12:32:39.904321] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-10 12:32:39.904353] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-10 12:32:39.905267] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 05:57:07.418180] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 05:57:07.419395] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 05:57:07.424314] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 05:57:07.424340] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 05:57:07.424601] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 05:57:07.424622] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 05:57:12.952890] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-11 05:57:12.953005] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 05:57:12.953046] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 05:57:12.953085] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:12.953123] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:12.953160] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:57:12.953198] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:12.953237] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:12.953274] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:12.953312] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:57:12.953350] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:12.953387] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:12.953425] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:12.953459] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:57:12.953501] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:12.953540] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:12.953606] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:12.953652] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:57:12.953689] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:12.953726] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:12.953767] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:12.953804] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:57:12.953841] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:12.953864] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:12.953875] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:12.953927] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:57:12.954510] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 05:57:13.496008] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 05:57:13.496138] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 05:57:13.537017] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 05:57:13.537084] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 05:57:13.578829] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 05:57:13.578873] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 05:57:13.617851] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 05:57:13.617905] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 05:57:13.625468] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 05:57:13.625527] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 05:57:13.665023] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 05:57:13.665086] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 05:57:13.705694] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 05:57:13.705745] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 05:57:13.713103] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 05:57:13.713150] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 05:57:13.751927] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 05:57:13.751971] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 05:57:13.791812] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 05:57:13.791850] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 05:57:13.791994] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.797791] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.835684] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.840945] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.846293] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.885852] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.938007] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.942659] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.947206] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 05:57:13.990641] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 05:57:13.994927] E [socket.c:2158:socket_connect_finish] 0-cinder-client-3: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:13.995078] E [socket.c:2158:socket_connect_finish] 0-cinder-client-5: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:13.995276] E [socket.c:2158:socket_connect_finish] 0-cinder-client-9: connection to 10.70.37.140:24007 failed (Connection refused)
>[2014-02-11 05:57:14.029841] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:14.030012] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-6: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:14.030088] I [client.c:2103:client_rpc_notify] 0-cinder-client-0: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:14.030121] I [client.c:2103:client_rpc_notify] 0-cinder-client-6: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:14.030145] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 05:57:14.030191] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 05:57:14.030235] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 05:57:14.084641] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:14.084779] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:14.084886] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:14.099797] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:14.099912] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 05:57:14.099984] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:14.100261] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 05:57:14.100647] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:14.100704] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 05:57:14.100739] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:14.100899] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 05:57:14.101113] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:14.101149] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 05:57:14.101226] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:14.101370] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 05:57:14.101559] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 05:57:14.101681] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 05:57:14.101823] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 05:57:17.455342] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 05:57:17.463333] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:17.463599] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:17.463619] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 05:57:17.463629] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:17.463687] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 05:57:17.463860] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 05:57:17.472188] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 05:57:17.480944] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:57:17.481641] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:57:17.481689] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 05:57:17.481709] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:57:17.481937] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 05:57:17.482203] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 05:57:18.955491] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:18.955684] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-7: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:57:18.955813] I [client.c:2103:client_rpc_notify] 0-cinder-client-1: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:18.956045] I [client.c:2103:client_rpc_notify] 0-cinder-client-7: disconnected from 10.70.37.77:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:57:18.996222] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 05:57:18.996986] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 05:57:18.997699] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-11 05:57:18.997779] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 05:57:18.997973] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 05:57:18.998122] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-11 05:57:18.998199] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 05:57:18.998505] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 05:57:19.003082] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 05:57:19.012687] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 05:57:19.020851] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2014-02-11 05:57:19.020982] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 05:57:19.021991] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 05:58:20.867188] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 05:58:20.869051] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 05:58:20.873127] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 05:58:20.873161] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 05:58:20.873319] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 05:58:20.873336] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 05:58:25.873890] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-11 05:58:25.873913] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 05:58:25.873924] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 05:58:25.873935] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.873945] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.873954] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 05:58:25.873964] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.873974] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.873984] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.873994] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 05:58:25.874005] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.874014] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.874024] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.874033] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 05:58:25.874043] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.874052] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.874062] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.874071] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 05:58:25.874081] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.874091] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.874100] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.874110] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 05:58:25.874120] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.874136] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.874147] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.874156] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 05:58:25.874350] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 05:58:25.880254] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 05:58:25.880304] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 05:58:25.882510] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 05:58:25.882532] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 05:58:25.883241] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 05:58:25.883261] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 05:58:25.883983] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 05:58:25.884002] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 05:58:25.884666] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 05:58:25.884690] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 05:58:25.885299] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 05:58:25.885321] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 05:58:25.885968] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 05:58:25.885992] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 05:58:25.886615] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 05:58:25.886640] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 05:58:25.887254] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 05:58:25.887276] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 05:58:25.887938] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 05:58:25.887958] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 05:58:25.888029] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.891577] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.894686] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.897827] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.900931] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.904052] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.907219] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.910389] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.913415] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 05:58:25.916502] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 05:58:25.920757] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 05:58:25.920845] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 05:58:25.927381] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 05:58:25.930621] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.930738] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.930839] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 05:58:25.930853] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 05:58:25.937253] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.937278] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 05:58:25.937290] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.937376] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 05:58:25.937442] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.937442] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 05:58:25.937449] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.937580] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 05:58:25.937690] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 05:58:25.937698] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 05:58:25.937952] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.938308] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.938528] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:25.939050] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.939072] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 05:58:25.939082] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.939143] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 05:58:25.939389] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.939409] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 05:58:25.939419] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.939478] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 05:58:25.939773] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:25.939793] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 05:58:25.939803] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:25.939893] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 05:58:25.939925] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 05:58:25.940185] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 05:58:25.940441] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 05:58:26.201110] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 05:58:26.207692] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 05:58:26.213680] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:26.213884] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 05:58:26.214254] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:26.214287] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 05:58:26.214298] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:26.214572] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 05:58:26.214672] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 05:58:26.214691] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 05:58:26.214918] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 05:58:26.215006] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 05:58:27.328894] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:27.329063] I [client.c:2103:client_rpc_notify] 0-cinder-client-3: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:27.329218] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:27.329300] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 05:58:27.329382] I [client.c:2103:client_rpc_notify] 0-cinder-client-5: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:27.329511] I [client.c:2103:client_rpc_notify] 0-cinder-client-9: disconnected from 10.70.37.140:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 05:58:27.340643] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 05:58:27.342205] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 05:58:27.342317] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 05:58:27.342459] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 05:58:27.342612] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-0: selecting local read_child cinder-client-0
>[2014-02-11 05:58:27.342657] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 05:58:27.343000] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 05:58:27.343161] I [afr-common.c:2274:afr_discovery_cbk] 0-cinder-replicate-3: selecting local read_child cinder-client-6
>[2014-02-11 05:58:27.347169] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 05:58:27.359413] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 05:58:27.367451] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 05:58:27.367498] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 05:58:27.368399] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:06:42.154415] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:06:42.155297] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 06:06:42.157763] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:06:42.157786] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:06:42.157975] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:06:42.158003] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:06:43.153770] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (Connection reset by peer)
>[2014-02-11 06:06:43.154134] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x164) [0x396600f344] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x396600ee83] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x396600ed9e]))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2014-02-11 06:06:42.165749 (xid=0x1x)
>[2014-02-11 06:06:43.154191] E [glusterfsd-mgmt.c:1470:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:cinder)
>[2014-02-11 06:06:43.154259] W [glusterfsd.c:1097:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x396600ed9e] (-->/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x20f) [0x396600ecaf] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3d6) [0x40b966]))) 0-: received signum (0), shutting down
>[2014-02-11 06:27:33.209411] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.44.1u2rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:27:33.211033] I [rpcsvc.c:2058:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
>[2014-02-11 06:27:33.215669] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:27:33.215702] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:27:33.215937] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:27:33.215962] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:27:38.968183] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-11 06:27:38.968220] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:27:38.968232] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:27:38.968243] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:38.968252] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:38.968262] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:27:38.968271] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:38.968282] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:38.968292] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:38.968301] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:27:38.968312] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:38.968321] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:38.968330] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:38.968340] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:27:38.968349] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:38.968359] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:38.968368] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:38.968377] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:27:38.968387] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:38.968396] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:38.968406] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:38.968415] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:27:38.968425] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:38.968434] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:38.968444] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:38.968458] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:27:38.968720] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:27:39.051654] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:27:39.051763] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:27:39.082646] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:27:39.082688] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:27:39.112404] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:27:39.112471] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:27:39.220440] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:27:39.220535] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:27:39.233738] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:27:39.233763] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:27:39.258477] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:27:39.270590] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:27:39.285795] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:27:39.285825] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:27:39.304514] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:27:39.304553] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:27:39.317275] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:27:39.317311] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:27:39.344973] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:27:39.345046] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:27:39.345173] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.363524] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.387584] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.404785] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.429123] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.445224] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.467332] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.504587] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.524531] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:27:39.539959] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:27:39.573697] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:27:39.573895] E [client-handshake.c:1759:client_query_portmap_cbk] 0-cinder-client-6: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:27:39.574088] I [client.c:2103:client_rpc_notify] 0-cinder-client-0: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:27:39.574169] I [client.c:2103:client_rpc_notify] 0-cinder-client-6: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:27:39.574425] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 06:27:39.574506] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:27:39.574551] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:27:39.650364] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 06:27:39.650452] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:27:39.650513] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:27:39.650566] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:27:39.650612] I [rpc-clnt.c:1689:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:27:39.753633] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.753781] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.760812] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.760911] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.760968] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.761033] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.761149] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.761246] I [client-handshake.c:1676:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:27:39.761377] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.761397] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 06:27:39.761418] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.761521] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-11 06:27:39.761697] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.761719] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:27:39.761729] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.761790] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 06:27:39.763931] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.763952] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:27:39.763962] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.764130] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.764148] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:27:39.764157] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.764230] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 06:27:39.764335] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.764360] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 06:27:39.764370] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.764517] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-11 06:27:39.764677] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 06:27:39.764713] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.764778] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:27:39.764805] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.765045] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.765074] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:27:39.765095] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.765181] I [afr-common.c:3885:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 06:27:39.765212] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:27:39.765239] I [client-handshake.c:1460:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:27:39.765251] I [client-handshake.c:1474:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:27:39.765260] I [client-handshake.c:1486:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:27:39.765542] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:27:39.765599] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:27:39.765640] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 06:27:39.776012] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:27:39.776416] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:27:39.776578] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:27:40.575628] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-0: added root inode
>[2014-02-11 06:27:40.577615] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-2: added root inode
>[2014-02-11 06:27:40.577731] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-4: added root inode
>[2014-02-11 06:27:40.577860] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-1: added root inode
>[2014-02-11 06:27:40.578361] I [afr-common.c:2211:afr_set_root_inode_on_first_lookup] 0-cinder-replicate-3: added root inode
>[2014-02-11 06:27:40.581817] I [dht-common.c:2645:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:27:40.593835] I [dht-rebalance.c:1116:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:27:40.611584] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-3: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-11 06:27:40.614805] I [dht-rebalance.c:1333:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.02 secs
>[2014-02-11 06:27:40.624301] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2014-02-11 06:27:40.624340] I [dht-rebalance.c:1771:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:27:40.625635] W [glusterfsd.c:1097:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-11 06:31:32.076787] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-11 06:31:32.082121] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-11 06:31:32.082166] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-11 06:31:32.082338] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-11 06:31:32.082357] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-11 06:31:38.089464] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-11 06:31:38.089491] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-11 06:31:38.089503] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-11 06:31:38.089515] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:38.089525] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:38.089535] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-11 06:31:38.089546] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:38.089560] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:38.089571] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:38.089582] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-11 06:31:38.089593] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:38.089603] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:38.089613] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:38.089624] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-11 06:31:38.089634] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:38.089644] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:38.089655] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:38.089688] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-11 06:31:38.089702] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:38.089712] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:38.089722] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:38.089752] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-11 06:31:38.089768] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:38.089778] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:38.089795] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:38.089805] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-11 06:31:38.090029] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-11 06:31:38.096556] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-11 06:31:38.096606] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-11 06:31:38.097280] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-11 06:31:38.097298] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-11 06:31:38.097993] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-11 06:31:38.098012] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-11 06:31:38.098634] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-11 06:31:38.098653] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-11 06:31:38.099337] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-11 06:31:38.099355] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-11 06:31:38.100041] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-11 06:31:38.100061] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-11 06:31:38.100708] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-11 06:31:38.100726] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-11 06:31:38.101396] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-11 06:31:38.101414] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-11 06:31:38.102109] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-11 06:31:38.102128] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-11 06:31:38.102823] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-11 06:31:38.102841] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-11 06:31:38.102912] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.106186] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.120420] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.123577] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.126599] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.129631] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.149976] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.153102] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.156140] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-11 06:31:38.170384] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-11 06:31:38.175559] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-11 06:31:38.175612] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-6: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-11 06:31:38.178844] I [client.c:2103:client_rpc_notify] 0-cinder-client-6: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-11 06:31:38.178882] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-11 06:31:38.178927] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-11 06:31:38.178958] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-11 06:31:38.179007] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-11 06:31:38.179047] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-11 06:31:38.206159] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-11 06:31:38.221945] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227301] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227422] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227534] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227594] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227655] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227744] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:38.227838] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.227862] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-11 06:31:38.227899] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.227997] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-11 06:31:38.228045] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.228082] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-11 06:31:38.228082] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.228170] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-11 06:31:38.228207] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.228207] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-11 06:31:38.228232] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.228428] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.228464] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-11 06:31:38.228464] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.228533] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-11 06:31:38.228611] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-11 06:31:38.228630] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-11 06:31:38.228745] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.228745] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-11 06:31:38.228757] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.228818] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-11 06:31:38.228852] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.228865] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-11 06:31:38.228874] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.229021] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:38.229038] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-11 06:31:38.229070] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:38.229235] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-11 06:31:38.229283] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-11 06:31:38.229301] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-11 06:31:38.229571] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-11 06:31:38.229884] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-11 06:31:42.123557] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-11 06:31:42.126934] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:42.127210] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:42.127230] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-11 06:31:42.127241] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:42.127300] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-11 06:31:42.127447] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-11 06:31:42.401666] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-11 06:31:42.405326] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-11 06:31:42.409359] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:42.418975] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:42.419013] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-11 06:31:42.419038] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:42.419241] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-11 06:31:42.419635] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-11 06:31:42.420773] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-11 06:31:42.420794] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-11 06:31:42.420805] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-11 06:31:42.421174] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-11 06:31:42.430246] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-11 06:31:42.442846] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-11 06:31:42.453469] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-11 06:31:42.460030] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-11 06:31:42.460054] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-11 06:31:42.460672] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 05:12:24.978417] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 05:12:24.985826] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 05:12:24.985926] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 05:12:24.986316] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 05:12:24.986350] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 05:12:30.011956] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 05:12:30.011980] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 05:12:30.011992] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 05:12:30.012003] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:30.012013] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:30.012034] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 05:12:30.012045] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:30.012057] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:30.012067] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:30.012077] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 05:12:30.012087] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:30.012096] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:30.012106] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:30.012115] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 05:12:30.012126] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:30.012135] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:30.012145] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:30.012155] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 05:12:30.012165] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:30.012174] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:30.012184] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:30.012193] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 05:12:30.012204] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:30.012213] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:30.012223] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:30.012232] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 05:12:30.012393] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-3
>[2014-02-12 05:12:30.012478] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 05:12:30.066341] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 05:12:30.066408] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 05:12:30.069409] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 05:12:30.069429] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 05:12:30.072046] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 05:12:30.072066] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 05:12:30.075515] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 05:12:30.075535] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 05:12:30.078079] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 05:12:30.078099] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 05:12:30.081464] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 05:12:30.081504] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 05:12:30.084866] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 05:12:30.084889] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 05:12:30.087588] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 05:12:30.087588] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 05:12:30.090892] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 05:12:30.090911] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 05:12:30.094123] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 05:12:30.094142] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 05:12:30.094208] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.097662] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.100762] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.103897] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.107076] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.110247] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.113440] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.116646] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.119757] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 05:12:30.122884] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.96
> 94:     option remote-subvolume /rhs/brick1/c11
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.77
>104:     option remote-subvolume /rhs/brick1/c12
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.121
>124:     option remote-subvolume /rhs/brick1/c1
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.140
>134:     option remote-subvolume /rhs/brick1/c2
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 5
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     option decommissioned-bricks cinder-replicate-3
>160:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>161: end-volume
>162: 
>163: volume cinder-write-behind
>164:     type performance/write-behind
>165:     subvolumes cinder-dht
>166: end-volume
>167: 
>168: volume cinder
>169:     type debug/io-stats
>170:     option latency-measurement off
>171:     option count-fop-hits off
>172:     subvolumes cinder-write-behind
>173: end-volume
>174: 
>+------------------------------------------------------------------------------+
>[2014-02-12 05:12:30.127585] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 05:12:30.130647] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49154 (from 0)
>[2014-02-12 05:12:30.134156] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:30.134263] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:30.134572] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:30.134594] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.96:49154, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 05:12:30.134604] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:30.134662] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 05:12:30.134702] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:30.134715] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 05:12:30.134724] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:30.134803] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 05:12:30.134959] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 05:12:30.134987] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 05:12:39.002643] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 05:12:39.002820] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 05:12:39.013833] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 05:12:39.023212] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.023578] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.023726] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.032574] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.032625] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 05:12:39.032645] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.032764] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 05:12:39.032833] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.032856] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 05:12:39.032874] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.033005] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 05:12:39.033070] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.033094] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 05:12:39.033156] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.033264] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 05:12:39.033351] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 05:12:39.033526] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 05:12:39.033778] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 05:12:39.130103] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 05:12:39.136263] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 05:12:39.142262] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 05:12:39.147128] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.147443] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.147576] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:39.147873] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.147899] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 05:12:39.147910] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.148110] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.148127] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 05:12:39.148169] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.148345] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:39.148364] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 05:12:39.148373] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:39.148528] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 05:12:39.148631] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 05:12:39.148881] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 05:12:42.160296] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 05:12:42.165714] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49154 (from 0)
>[2014-02-12 05:12:42.174140] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:42.174305] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 05:12:42.174631] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:42.174661] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 05:12:42.174678] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:42.175023] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 05:12:42.175073] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 05:12:42.175092] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.77:49154, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 05:12:42.175109] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 05:12:42.175480] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 05:12:42.210021] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 05:12:42.214123] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 05:12:42.244305] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80: attempting to move from cinder-replicate-3 to cinder-replicate-4
>[2014-02-12 05:14:57.668346] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80 from subvolume cinder-replicate-3 to cinder-replicate-4
>[2014-02-12 05:14:57.672224] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-f273f2f9-5d12-4496-b4d2-cb5272db03e9: attempting to move from cinder-replicate-3 to cinder-replicate-4
>[2014-02-12 05:19:22.534144] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-f273f2f9-5d12-4496-b4d2-cb5272db03e9 from subvolume cinder-replicate-3 to cinder-replicate-4
>[2014-02-12 05:19:22.540079] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 400.33 secs
>[2014-02-12 05:19:22.545757] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 400.00 secs
>[2014-02-12 05:19:22.545757] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 21474836480, lookups: 6, failures: 0, skipped: 0
>[2014-02-12 05:19:22.553841] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:40:36.080279] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:40:36.086637] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:40:36.086716] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:40:36.087276] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:40:36.087336] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:40:41.091565] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 06:40:41.091609] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-12 06:40:41.091631] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:40:41.091652] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:41.091671] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:41.091690] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:41.091710] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:41.091731] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:41.091801] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:41.091823] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:41.091844] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:41.091863] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:41.091883] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:41.091922] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:41.091943] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:41.091962] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:41.091981] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:41.091999] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:41.092019] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:41.092037] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:41.092055] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:41.092074] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:41.092093] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:41.092111] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:41.092129] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:41.092153] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:41.092647] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:40:41.153883] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:40:41.153938] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:40:41.157995] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:40:41.158016] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:40:41.162047] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:40:41.162068] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:40:41.166261] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:40:41.166283] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:40:41.170212] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:40:41.170232] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:40:41.174236] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:40:41.174256] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:40:41.178205] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:40:41.178224] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:40:41.182253] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:40:41.182275] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:40:41.186229] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:40:41.186277] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:40:41.190533] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:40:41.190556] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:40:41.190622] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.194428] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.197659] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.200825] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.204018] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.207144] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.210459] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.213655] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.216805] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:41.220089] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 4
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:40:41.225004] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:40:41.228303] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:40:41.231806] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:41.231901] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:41.232202] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:41.232224] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:40:41.232234] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:41.232292] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:40:41.232343] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:41.232358] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:40:41.232368] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:41.232450] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 06:40:41.232605] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:40:41.232633] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:40:46.347933] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:40:46.353234] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:40:46.353292] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:40:46.360590] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.360979] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.362121] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.362148] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:40:46.362158] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.362250] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 06:40:46.362332] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:40:46.366455] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.366503] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:40:46.366521] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.366616] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:40:46.366651] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:40:46.366673] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.367064] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:40:46.370696] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.370824] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:40:46.370871] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:40:46.370910] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:40:46.381002] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.381052] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:40:46.381063] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.381353] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:40:46.381370] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.381383] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:40:46.381400] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.381555] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 06:40:46.381776] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:40:46.381894] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.382114] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:40:46.382202] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.382334] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.382687] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:46.382792] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.382810] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:40:46.382820] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.382992] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.383011] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:40:46.383021] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.383184] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:40:46.383477] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.383563] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:40:46.383577] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.383805] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:40:46.383944] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:46.383965] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:40:46.383999] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:46.384328] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:40:46.384428] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:40:46.413095] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:40:46.425070] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:40:46.425117] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-12 06:40:46.425828] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:40:50.650872] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:40:50.656523] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:40:50.656621] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:40:50.656924] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:40:50.656956] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:40:55.652512] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 06:40:55.652535] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 06:40:55.652546] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:40:55.652557] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:55.652567] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:55.652587] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:40:55.652598] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:55.652609] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:55.652620] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:55.652630] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:40:55.652640] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:55.652650] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:55.652660] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:55.652670] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:40:55.652680] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:55.652690] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:55.652700] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:55.652710] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:40:55.652720] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:55.652730] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:55.652765] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:55.652776] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:40:55.652786] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:55.652796] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:55.652806] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:55.652815] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:40:55.653018] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:40:55.660197] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:40:55.705124] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:40:55.706425] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:40:55.706462] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:40:55.707680] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:40:55.707715] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:40:55.708927] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:40:55.708961] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:40:55.710141] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:40:55.710175] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:40:55.711392] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:40:55.711425] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:40:55.712234] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:40:55.712253] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:40:55.712973] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:40:55.712991] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:40:55.713807] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:40:55.713827] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:40:55.717836] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:40:55.717861] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:40:55.717931] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.721521] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.724764] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.727945] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.731087] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.734252] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.737389] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.740600] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.743812] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:40:55.746956] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-dht
>152:     type cluster/distribute
>153:     option use-readdirp yes
>154:     option lookup-unhashed yes
>155:     option assert-no-child-down yes
>156:     option readdir-optimize on
>157:     option rebalance-cmd 1
>158:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>159:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4
>160: end-volume
>161: 
>162: volume cinder-write-behind
>163:     type performance/write-behind
>164:     subvolumes cinder-dht
>165: end-volume
>166: 
>167: volume cinder
>168:     type debug/io-stats
>169:     option latency-measurement off
>170:     option count-fop-hits off
>171:     subvolumes cinder-write-behind
>172: end-volume
>173: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:40:55.752994] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:40:55.753049] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:40:55.759774] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:55.759867] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:40:55.760175] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:55.760195] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:40:55.760206] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:55.760264] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:40:55.760305] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:40:55.760318] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:40:55.760328] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:40:55.760388] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 06:40:55.760417] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:40:55.760594] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:41:00.865373] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:41:00.866368] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:41:00.866480] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:41:00.884437] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.884951] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.885329] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.885350] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:41:00.885371] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.885439] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:41:00.886139] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.886159] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:41:00.886169] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.886227] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 06:41:00.886612] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:41:00.887179] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:41:00.887658] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.888210] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.888232] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:41:00.888242] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.888315] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 06:41:00.888682] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:41:00.892827] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:41:00.896610] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:41:00.896685] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:41:00.905090] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:41:00.905155] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:41:00.912686] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.912945] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.913145] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.913548] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.913808] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.913844] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:41:00.913868] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.914274] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.914307] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:41:00.914318] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.914587] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:41:00.914692] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:41:00.914905] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.914935] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:41:00.914957] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.915256] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.915283] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:41:00.915336] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.915469] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:41:00.915509] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:41:00.915523] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:41:00.915559] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:41:00.916427] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:41:00.916537] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:41:00.916575] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:41:00.940565] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:41:00.944699] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 06:41:00.958927] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.01 secs
>[2014-02-12 06:41:00.964643] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:41:00.964676] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 06:41:00.965282] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:51:29.529586] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=4 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:51:29.534191] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:51:29.534268] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:51:29.534525] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:51:29.534557] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:51:34.537492] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 06:51:34.537514] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '4'
>[2014-02-12 06:51:34.537526] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:51:34.537537] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:34.537548] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:34.537557] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:34.537567] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:34.537578] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:34.537588] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:34.537598] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:34.537609] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:34.537619] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:34.537629] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:34.537638] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:34.537649] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:34.537658] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:34.537668] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:34.537678] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:34.537688] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:34.537698] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:34.537715] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:34.537730] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:34.537769] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:34.537780] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:34.537790] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:34.537799] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:34.537810] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:34.537819] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:34.537829] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:34.537839] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:34.538046] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:51:34.545861] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 06:51:34.545925] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 06:51:34.546563] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 06:51:34.546581] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 06:51:34.547249] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:51:34.547267] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:51:34.547925] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:51:34.547943] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:51:34.548633] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:51:34.548652] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:51:34.549394] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:51:34.549413] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:51:34.550143] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:51:34.550161] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:51:34.550900] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:51:34.550919] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:51:34.551679] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:51:34.551708] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:51:34.552593] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:51:34.552614] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:51:34.553311] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:51:34.553341] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:51:34.554050] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:51:34.554068] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:51:34.554140] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.557660] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.560781] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.564056] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.567158] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.570235] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.573509] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.576564] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.579652] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.582975] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.586071] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:34.589225] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 4
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:51:34.593922] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:51:34.597147] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:51:34.600372] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 06:51:34.604246] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:34.604553] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:34.604625] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:34.604709] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:34.604728] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:51:34.604769] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:34.604833] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:51:34.604875] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:34.604888] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:51:34.604897] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:34.604959] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 06:51:34.604987] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:51:34.605027] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:51:34.605257] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:34.605275] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 06:51:34.605285] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:34.605340] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 06:51:34.605516] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 06:51:39.720161] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:51:39.724435] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:51:39.724514] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:51:39.731588] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.732083] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.732478] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.732544] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:51:39.732563] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.732776] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 06:51:39.732832] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.733150] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.733177] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:51:39.733194] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.733312] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 06:51:39.733520] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.733586] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:51:39.733606] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.733776] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 06:51:39.733832] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:51:39.733943] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:51:39.734230] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:51:39.735748] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:51:39.739736] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:51:39.739808] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:51:39.746893] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.747019] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.747117] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.747380] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.747399] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:51:39.747422] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.747667] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.747687] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:51:39.747696] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.747902] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.747921] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:51:39.747931] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.748116] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:51:39.748232] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:51:39.748537] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:51:39.762095] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:51:39.765736] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:51:39.765855] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 06:51:39.773599] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.773880] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.774739] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:39.775116] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.775139] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:51:39.775149] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.775726] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.775749] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 06:51:39.775760] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.776000] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:51:39.776881] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:39.776908] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:51:39.776928] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:39.777221] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 06:51:39.777685] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:51:39.786400] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:51:39.802892] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 06:51:39.802958] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
>[2014-02-12 06:51:39.803568] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:51:48.290257] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:51:48.296342] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:51:48.296412] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:51:48.296664] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:51:48.296713] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:51:53.324035] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 06:51:53.324061] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 06:51:53.324073] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:51:53.324083] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:53.324093] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:53.324103] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:51:53.324113] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:53.324124] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:53.324134] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:53.324144] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:51:53.324167] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:53.324178] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:53.324188] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:53.324198] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:51:53.324208] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:53.324218] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:53.324227] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:53.324237] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:51:53.324248] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:53.324258] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:53.324267] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:53.324283] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:51:53.324293] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:53.324303] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:53.324313] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:53.324323] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:51:53.324333] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:53.324343] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:53.324352] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:53.324362] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:51:53.324699] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:51:53.342129] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 06:51:53.342188] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 06:51:53.343485] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 06:51:53.343505] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 06:51:53.344691] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:51:53.344722] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:51:53.346227] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:51:53.346247] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:51:53.347514] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:51:53.347533] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:51:53.348244] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:51:53.348263] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:51:53.349185] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:51:53.349207] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:51:53.350583] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:51:53.350603] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:51:53.352500] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:51:53.352523] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:51:53.353973] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:51:53.353994] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:51:53.354955] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:51:53.354975] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:51:53.355682] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:51:53.355702] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:51:53.355801] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.359345] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.362583] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.365712] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.368868] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.372135] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.375357] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.378542] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.381713] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.384992] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.388145] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 06:51:53.391415] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 1
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:51:53.396145] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:51:53.396222] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 06:51:53.402699] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 06:51:53.406356] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:53.406449] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:53.406528] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:53.406833] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:53.406855] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:51:53.406866] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:53.406962] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 06:51:53.407004] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:53.407017] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 06:51:53.407027] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:53.407087] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 06:51:53.407122] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:51:53.407146] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:53.407158] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 06:51:53.407168] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:53.407246] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 06:51:53.407434] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 06:51:53.407462] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 06:51:58.539032] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:51:58.568878] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:51:58.575982] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:51:58.576132] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:51:58.576196] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:51:58.576254] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:51:58.589910] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:51:58.589974] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:51:58.590066] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 06:51:58.590184] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.600412] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.600507] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:51:58.600527] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.600594] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-12 06:51:58.600675] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.600907] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601077] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601159] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601225] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601284] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:51:58.601391] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601466] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.601507] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:51:58.601523] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.601581] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 06:51:58.601696] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.601810] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.601828] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:51:58.601838] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.602047] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.602066] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:51:58.602075] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.602172] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-12 06:51:58.602208] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.602221] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:51:58.602230] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.602424] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:51:58.602458] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.602470] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 06:51:58.602480] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.602800] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.602842] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:51:58.602863] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.603019] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:51:58.603070] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:51:58.603096] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 06:51:58.603148] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.603183] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:51:58.603195] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.603377] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:51:58.603406] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:51:58.603588] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:51:58.603623] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:51:58.603850] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:51:58.603869] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:51:58.603879] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:51:58.604263] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:51:58.613785] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:51:58.618796] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 06:51:58.629807] I [dht-common.c:1036:dht_lookup_everywhere_cbk] 0-cinder-dht: deleting stale linkfile /volume-6233b63b-8d29-496f-94a5-f75af023b0c4 on cinder-replicate-4
>[2014-02-12 06:51:58.650291] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80: attempting to move from cinder-replicate-4 to cinder-replicate-2
>[2014-02-12 06:54:48.280837] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
>[2014-02-12 06:54:52.480038] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 06:58:37.384275] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 06:58:37.388573] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 06:58:37.388617] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 06:58:37.388808] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 06:58:37.388823] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 06:58:54.906069] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 06:58:54.906089] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 06:58:54.906097] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 06:58:54.906105] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:58:54.906111] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:58:54.906118] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 06:58:54.906125] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:58:54.906133] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:58:54.906140] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:58:54.906147] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 06:58:54.906154] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:58:54.906161] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:58:54.906167] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:58:54.906174] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 06:58:54.906198] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:58:54.906209] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:58:54.906216] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:58:54.906222] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 06:58:54.906230] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:58:54.906236] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:58:54.906243] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:58:54.906249] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 06:58:54.906272] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:58:54.906280] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:58:54.906287] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:58:54.906293] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 06:58:54.906300] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:58:54.906306] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:58:54.906313] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:58:54.906319] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 06:58:54.906560] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 06:58:56.324585] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 06:58:56.324612] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 06:58:56.359468] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 06:58:56.359513] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 06:58:57.565481] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 06:58:57.565528] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 06:58:57.567132] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 06:58:57.567145] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 06:58:57.568983] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 06:58:57.568996] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 06:58:57.570608] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 06:58:57.570621] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 06:58:57.572371] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 06:58:57.572391] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 06:58:57.574318] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 06:58:57.574331] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 06:58:57.603364] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 06:58:57.603421] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 06:58:57.605649] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 06:58:57.605665] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 06:58:57.607533] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 06:58:57.607548] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 06:58:57.610279] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 06:58:57.610303] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 06:58:57.610368] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:57.613073] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:57.648631] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:57.651164] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:57.653682] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:57.656221] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.914801] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.918627] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.921123] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.923649] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.956086] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 06:58:58.958633] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 1
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 06:58:58.962324] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 06:58:58.962372] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 06:58:58.962412] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 06:59:00.171921] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 06:59:00.171978] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 06:59:00.172009] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 06:59:00.172063] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 06:59:00.217764] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 06:59:00.217833] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 06:59:01.413907] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.414022] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.414102] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.414154] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 06:59:01.414219] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 06:59:01.414269] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-10: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 06:59:01.416903] I [client.c:2103:client_rpc_notify] 0-cinder-client-8: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-12 06:59:01.416960] I [client.c:2103:client_rpc_notify] 0-cinder-client-10: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-12 06:59:01.417030] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.417080] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.417119] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.417157] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.417261] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.417290] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 06:59:01.417327] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.417361] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-3' came back up; going online.
>[2014-02-12 06:59:01.417388] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.417396] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 06:59:01.417403] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.417437] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 06:59:01.417457] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.417495] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.417515] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 06:59:01.417523] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.417687] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.417705] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 06:59:01.417720] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.417773] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 06:59:01.446648] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.446685] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 06:59:01.446694] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.446784] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-7' came back up; going online.
>[2014-02-12 06:59:01.446815] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 06:59:01.446834] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.446899] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 06:59:01.446926] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 06:59:01.446948] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.446956] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 06:59:01.446963] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.447022] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 06:59:01.447104] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.447138] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 06:59:01.447147] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.447278] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 06:59:01.447305] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.447315] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 06:59:01.447322] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.447494] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:01.447569] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 06:59:01.447656] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.447678] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 06:59:01.447690] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.447795] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-11' came back up; going online.
>[2014-02-12 06:59:01.447926] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 06:59:01.447958] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 06:59:01.448264] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 06:59:01.448286] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 06:59:01.458539] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:01.458560] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 06:59:01.458567] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:01.468809] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 06:59:02.693457] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 06:59:02.710884] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 06:59:02.734438] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 06:59:02.740007] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 06:59:02.748720] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 06:59:02.748752] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 06:59:02.748779] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 06:59:02.749043] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 06:59:03.948438] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-4: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-12 06:59:03.950028] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-4: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-12 06:59:03.953623] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 1.24 secs
>[2014-02-12 06:59:03.970902] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 2.00 secs
>[2014-02-12 06:59:03.970960] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 06:59:03.971674] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 07:01:12.659563] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 07:01:12.669859] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 07:01:12.669903] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 07:01:12.670050] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 07:01:12.670065] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 07:01:19.297106] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 07:01:19.297131] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 07:01:19.297142] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 07:01:19.297152] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:01:19.297161] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:01:19.297170] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:01:19.297179] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:01:19.297189] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:01:19.297199] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for[2014-02-12 07:02:59.496509] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 07:02:59.506440] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 07:02:59.506482] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 07:02:59.506636] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 07:02:59.506650] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 07:03:05.594868] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 07:03:05.594896] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '1'
>[2014-02-12 07:03:05.594907] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 07:03:05.594916] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:03:05.594924] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:03:05.594933] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 07:03:05.594941] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:03:05.594950] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:03:05.594959] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:03:05.594968] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 07:03:05.594977] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 07:03:05.594985] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 07:03:05.594993] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 07:03:05.595001] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 07:03:05.595010] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 07:03:05.595018] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 07:03:05.595026] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 07:03:05.595042] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 07:03:05.595052] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 07:03:05.595060] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 07:03:05.595068] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 07:03:05.595080] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 07:03:05.595089] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 07:03:05.595097] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 07:03:05.595106] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 07:03:05.595114] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 07:03:05.595122] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 07:03:05.595131] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 07:03:05.595139] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 07:03:05.595147] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 07:03:05.596486] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 07:03:05.701125] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 07:03:05.701181] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 07:03:05.702698] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 07:03:05.702715] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 07:03:05.704185] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 07:03:05.704201] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 07:03:05.705668] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 07:03:05.705683] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 07:03:05.746104] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 07:03:05.759514] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 07:03:05.761143] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 07:03:05.761164] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 07:03:05.762944] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 07:03:05.762965] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 07:03:05.764544] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 07:03:05.764570] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 07:03:05.766210] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 07:03:05.766227] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 07:03:05.767752] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 07:03:05.767769] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 07:03:05.769286] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 07:03:05.769304] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 07:03:05.784887] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 07:03:05.798281] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 07:03:05.798389] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.801637] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.804840] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.807916] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.822709] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.825857] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.828944] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.857007] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.860111] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.863253] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.893729] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 07:03:05.896930] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 1
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 07:03:05.913028] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 07:03:05.947057] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 07:03:05.947099] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 07:03:05.947124] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 07:03:05.947155] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 07:03:05.947188] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 07:03:05.966355] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 07:03:05.966461] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 07:03:05.966521] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 07:03:05.977582] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.977768] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.977830] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.977900] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.977968] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.978153] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.978300] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.978315] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 07:03:05.978324] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.978389] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 07:03:05.978425] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.978436] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 07:03:05.978444] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.978488] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 07:03:05.978551] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.978565] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 07:03:05.978593] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.978718] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 07:03:05.978749] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 07:03:05.978771] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.978780] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 07:03:05.978788] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.978838] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 07:03:05.978903] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.978918] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 07:03:05.978925] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.978971] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 07:03:05.979017] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:05.979030] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 07:03:05.979038] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:05.979103] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-11' came back up; going online.
>[2014-02-12 07:03:05.979182] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 07:03:05.979231] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 07:03:05.979267] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.979368] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 07:03:05.979411] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:05.979490] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 07:03:05.979509] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 07:03:05.980149] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:06.009737] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.009760] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 07:03:06.009770] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.009918] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.009934] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 07:03:06.009942] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.021024] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 07:03:06.021056] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 07:03:06.021084] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.021095] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 07:03:06.021103] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.021735] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 07:03:06.202909] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 07:03:06.206237] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 07:03:06.209446] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 07:03:06.212591] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:06.223804] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:06.223881] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 07:03:06.236824] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.236857] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 07:03:06.236866] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.237025] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.237041] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 07:03:06.237064] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.237195] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 07:03:06.237211] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 07:03:06.237219] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 07:03:06.245086] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 07:03:06.245119] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 07:03:06.245584] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 07:03:06.296570] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 07:03:06.390627] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 07:03:06.440965] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80: attempting to move from cinder-replicate-4 to cinder-replicate-2
>[2014-02-12 07:06:17.340706] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 191.00 secs
>[2014-02-12 07:06:17.340753] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:06:30.524640] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 204.00 secs
>[2014-02-12 07:06:30.524685] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:12:22.411864] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 556.00 secs
>[2014-02-12 07:12:22.411926] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:14:57.938876] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 711.00 secs
>[2014-02-12 07:14:57.938917] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:17:27.614990] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 861.00 secs
>[2014-02-12 07:17:27.615028] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:19:02.390945] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 956.00 secs
>[2014-02-12 07:19:02.390978] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:20:25.649499] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1039.00 secs
>[2014-02-12 07:20:25.649553] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 3, failures: 0, skipped: 0
>[2014-02-12 07:28:28.335512] W [dht-common.c:1022:dht_lookup_everywhere_cbk] 0-cinder-dht: multiple subvolumes (cinder-replicate-2 and cinder-replicate-3) have file /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80 (preferably rename the file in the backend, and do a fresh lookup)
>[2014-02-12 07:28:28.335890] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-7821d8df-bd77-45c0-b35f-e29573ec0b80 from subvolume cinder-replicate-4 to cinder-replicate-2
>[2014-02-12 07:28:28.341350] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-f273f2f9-5d12-4496-b4d2-cb5272db03e9: attempting to move from cinder-replicate-4 to cinder-replicate-2
>[2014-02-12 07:30:50.905993] I [dht-rebalance.c:881:dht_migrate_file] 0-cinder-dht: completed migration of /volume-f273f2f9-5d12-4496-b4d2-cb5272db03e9 from subvolume cinder-replicate-4 to cinder-replicate-2
>[2014-02-12 07:30:50.909985] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 1664.52 secs
>[2014-02-12 07:30:50.933512] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1664.00 secs
>[2014-02-12 07:30:50.933684] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 2, size: 21474836480, lookups: 4, failures: 0, skipped: 0
>[2014-02-12 07:30:50.962770] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 10:39:34.488022] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 10:39:34.493729] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 10:39:34.493774] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 10:39:34.494343] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 10:39:34.494365] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 10:39:39.508535] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 10:39:39.508560] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 10:39:39.508573] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 10:39:39.508584] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:39.508595] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:39.508606] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 10:39:39.508616] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:39.508628] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:39.508639] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:39.508652] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 10:39:39.508674] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:39.508711] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:39.508725] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:39.508736] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 10:39:39.508747] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:39.508757] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:39.508767] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:39.508777] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 10:39:39.508788] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:39.508798] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:39.508808] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:39.508818] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 10:39:39.508841] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:39.508851] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:39.508861] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:39.508871] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 10:39:39.508882] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:39.508891] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:39.508901] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:39.508911] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 10:39:39.509612] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 10:39:39.593182] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 10:39:39.593224] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 10:39:39.597615] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 10:39:39.597639] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 10:39:39.602045] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 10:39:39.602081] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 10:39:39.606075] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 10:39:39.606125] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 10:39:39.610053] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 10:39:39.610077] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 10:39:39.614202] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 10:39:39.614225] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 10:39:39.618468] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 10:39:39.618493] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 10:39:39.623614] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 10:39:39.623660] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 10:39:39.627601] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 10:39:39.627643] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 10:39:39.632364] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 10:39:39.632445] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 10:39:39.636598] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 10:39:39.636642] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 10:39:39.641451] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 10:39:39.641550] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 10:39:39.641643] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.645012] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.648238] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.651770] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.655849] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.659077] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.662533] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.665775] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.669083] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.672224] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.675326] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 10:39:39.678707] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>190: end-volume
>191: 
>192: volume cinder-write-behind
>193:     type performance/write-behind
>194:     subvolumes cinder-dht
>195: end-volume
>196: 
>197: volume cinder
>198:     type debug/io-stats
>199:     option latency-measurement off
>200:     option count-fop-hits off
>201:     subvolumes cinder-write-behind
>202: end-volume
>203: 
>+------------------------------------------------------------------------------+
>[2014-02-12 10:39:39.684735] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 10:39:39.684883] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 10:39:39.684966] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 10:39:39.685040] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 10:39:39.685201] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 10:39:39.685247] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 10:39:39.685285] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 10:39:39.685322] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 10:39:39.714446] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 10:39:39.714544] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 10:39:39.720936] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 10:39:39.720991] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 10:39:39.728503] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.728640] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.728755] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.729039] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.729070] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 10:39:39.729092] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.729267] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 10:39:39.729329] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.729577] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.729718] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.729933] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.730047] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.730268] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 10:39:39.730379] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.730667] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 10:39:39.730726] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.730858] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 10:39:39.730917] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.730944] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 10:39:39.731001] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.731149] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 10:39:39.731202] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.731563] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.731734] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.731765] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 10:39:39.731840] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.732018] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 10:39:39.732078] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.732104] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 10:39:39.732122] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.732715] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 10:39:39.733070] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.733132] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 10:39:39.733181] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.733452] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 10:39:39.733579] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.733808] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.733859] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 10:39:39.733884] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.734154] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 10:39:39.734239] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 10:39:39.734344] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.734436] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 10:39:39.734497] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.734717] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.734782] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 10:39:39.734807] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.734992] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 10:39:39.735081] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 10:39:39.735232] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.735260] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 10:39:39.735306] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.735550] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 10:39:39.735607] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.735623] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 10:39:39.735633] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.735764] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 10:39:39.735912] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 10:39:39.735990] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 10:39:39.736045] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 10:39:39.736065] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 10:39:39.736075] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 10:39:39.745495] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 10:39:39.745590] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 10:39:39.745794] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 10:39:39.745889] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 10:39:39.793903] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 10:39:39.802415] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 10:39:39.828250] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.03 secs
>[2014-02-12 10:39:39.841118] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 0.00 secs
>[2014-02-12 10:39:39.841269] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 5, failures: 0, skipped: 0
>[2014-02-12 10:39:39.841994] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 11:08:20.533088] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 11:08:20.539576] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 11:08:20.539723] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 11:08:20.540168] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 11:08:20.540255] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 11:08:25.586118] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 11:08:25.586142] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 11:08:25.586154] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 11:08:25.586165] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:25.586176] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:25.586186] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:08:25.586196] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:25.586213] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:25.586224] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:25.586235] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:08:25.586246] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:25.586256] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:25.586266] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:25.586277] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:08:25.586287] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:25.586297] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:25.586308] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:25.586318] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:08:25.586328] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:25.586338] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:25.586428] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:25.586519] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:08:25.586535] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:25.586545] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:25.586564] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:25.586575] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:08:25.586624] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:25.586641] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:25.586651] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:25.586662] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:08:25.586920] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-5
>[2014-02-12 11:08:25.587039] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 11:08:25.687879] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 11:08:25.687950] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 11:08:25.692592] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 11:08:25.692637] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 11:08:25.697435] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 11:08:25.697478] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 11:08:25.701477] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 11:08:25.701498] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 11:08:25.705885] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 11:08:25.705906] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 11:08:25.709972] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 11:08:25.709993] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 11:08:25.714342] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 11:08:25.714364] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 11:08:25.718464] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 11:08:25.718485] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 11:08:25.722505] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 11:08:25.722526] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 11:08:25.726519] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 11:08:25.726540] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 11:08:25.730505] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 11:08:25.730525] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 11:08:25.734427] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 11:08:25.734448] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 11:08:25.734526] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.737842] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.741149] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.744243] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.748374] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.754052] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.758030] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.761062] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.764093] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.767261] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.770422] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 11:08:25.773465] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     option decommissioned-bricks cinder-replicate-5
>190:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>191: end-volume
>192: 
>193: volume cinder-write-behind
>194:     type performance/write-behind
>195:     subvolumes cinder-dht
>196: end-volume
>197: 
>198: volume cinder
>199:     type debug/io-stats
>200:     option latency-measurement off
>201:     option count-fop-hits off
>202:     subvolumes cinder-write-behind
>203: end-volume
>204: 
>+------------------------------------------------------------------------------+
>[2014-02-12 11:08:25.780008] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-0: changing port to 49152 (from 0)
>[2014-02-12 11:08:25.784980] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-8: changing port to 49158 (from 0)
>[2014-02-12 11:08:25.785188] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-10: changing port to 49159 (from 0)
>[2014-02-12 11:08:25.788908] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 11:08:25.792103] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 11:08:25.792163] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 11:08:25.792209] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 11:08:25.792241] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 11:08:25.807936] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 11:08:25.807985] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.808089] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 11:08:25.808136] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 11:08:25.808168] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 11:08:25.820919] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.821016] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-0: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.821036] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-0: Connected to 10.70.37.96:49152, attached to remote volume '/rhs/brick1/c3'.
>[2014-02-12 11:08:25.821046] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.821133] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-0' came back up; going online.
>[2014-02-12 11:08:25.821172] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.821355] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-0: Server lk version = 1
>[2014-02-12 11:08:25.821448] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.821805] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-10: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.821826] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-10: Connected to 10.70.37.96:49159, attached to remote volume '/rhs/brick1/c7'.
>[2014-02-12 11:08:25.821837] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.821894] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-10' came back up; going online.
>[2014-02-12 11:08:25.821935] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822007] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822070] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822122] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822178] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822255] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-8: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.822273] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-8: Connected to 10.70.37.96:49158, attached to remote volume '/rhs/brick1/c11'.
>[2014-02-12 11:08:25.822283] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.822366] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-8' came back up; going online.
>[2014-02-12 11:08:25.822399] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822461] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822531] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:08:25.822600] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.822617] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 11:08:25.822627] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.822857] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.822878] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 11:08:25.822888] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.822982] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-5' came back up; going online.
>[2014-02-12 11:08:25.823017] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 11:08:25.823038] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-8: Server lk version = 1
>[2014-02-12 11:08:25.823063] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823075] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 11:08:25.823085] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.823139] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 11:08:25.823173] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823193] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 11:08:25.823204] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.823346] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823365] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 11:08:25.823375] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.823527] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823545] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 11:08:25.823555] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.823752] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 11:08:25.823781] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 11:08:25.823800] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-10: Server lk version = 1
>[2014-02-12 11:08:25.823825] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823838] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 11:08:25.823847] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.823904] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 11:08:25.823937] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.823950] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 11:08:25.823960] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.824088] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:08:25.824106] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 11:08:25.824116] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:08:25.829334] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 11:08:25.829445] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 11:08:25.829472] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 11:08:25.829492] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 11:08:25.829516] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 11:08:25.829554] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 11:08:25.860790] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 11:08:25.867823] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 11:08:25.893411] I [dht-rebalance.c:672:dht_migrate_file] 0-cinder-dht: /volume-6299fd6a-ff07-47c7-9e3d-c03def66a327: attempting to move from cinder-replicate-5 to cinder-replicate-4
>[2014-02-12 11:08:44.292742] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 19.00 secs
>[2014-02-12 11:08:44.298030] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 6, failures: 0, skipped: 0
>[2014-02-12 11:08:44.801218] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
>[2014-02-12 11:08:48.760104] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down
>[2014-02-12 11:09:19.650688] I [glusterfsd.c:2026:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.59rhs (/usr/sbin/glusterfs -s localhost --volfile-id cinder --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *replicate*.readdir-failover=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=304c4df7-e2a3-4d55-aec8-dc645694903e --socket-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.sock --pid-file /var/lib/glusterd/vols/cinder/rebalance/304c4df7-e2a3-4d55-aec8-dc645694903e.pid -l /var/log/glusterfs/cinder-rebalance.log)
>[2014-02-12 11:09:19.657253] I [socket.c:3505:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-02-12 11:09:19.657297] I [socket.c:3520:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-02-12 11:09:19.657466] I [socket.c:3505:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-02-12 11:09:19.657484] I [socket.c:3520:socket_init] 0-glusterfs: using system polling thread
>[2014-02-12 11:09:25.802378] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'node-uuid' for volume 'cinder-dht' with value '304c4df7-e2a3-4d55-aec8-dc645694903e'
>[2014-02-12 11:09:25.802425] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'rebalance-cmd' for volume 'cinder-dht' with value '5'
>[2014-02-12 11:09:25.802438] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'readdir-optimize' for volume 'cinder-dht' with value 'on'
>[2014-02-12 11:09:25.802450] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'assert-no-child-down' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:09:25.802454] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'lookup-unhashed' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:09:25.802474] I [graph.c:239:gf_add_cmdline_options] 0-cinder-dht: adding option 'use-readdirp' for volume 'cinder-dht' with value 'yes'
>[2014-02-12 11:09:25.802483] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'readdir-failover' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:09:25.802501] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'entry-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:09:25.802511] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'metadata-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:09:25.802521] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-5: adding option 'data-self-heal' for volume 'cinder-replicate-5' with value 'off'
>[2014-02-12 11:09:25.802545] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'readdir-failover' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:09:25.802552] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'entry-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:09:25.802567] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'metadata-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:09:25.802579] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-4: adding option 'data-self-heal' for volume 'cinder-replicate-4' with value 'off'
>[2014-02-12 11:09:25.802586] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'readdir-failover' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:09:25.802597] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'entry-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:09:25.802606] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'metadata-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:09:25.802614] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-3: adding option 'data-self-heal' for volume 'cinder-replicate-3' with value 'off'
>[2014-02-12 11:09:25.802615] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'readdir-failover' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:09:25.802626] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'entry-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:09:25.802636] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'metadata-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:09:25.802649] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-2: adding option 'data-self-heal' for volume 'cinder-replicate-2' with value 'off'
>[2014-02-12 11:09:25.802665] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'readdir-failover' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:09:25.802683] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'entry-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:09:25.802697] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'metadata-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:09:25.802715] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-1: adding option 'data-self-heal' for volume 'cinder-replicate-1' with value 'off'
>[2014-02-12 11:09:25.802730] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'readdir-failover' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:09:25.802759] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'entry-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:09:25.802775] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'metadata-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:09:25.802824] I [graph.c:239:gf_add_cmdline_options] 0-cinder-replicate-0: adding option 'data-self-heal' for volume 'cinder-replicate-0' with value 'off'
>[2014-02-12 11:09:25.802998] I [dht-shared.c:246:dht_parse_decommissioned_bricks] 0-cinder-dht: decommissioning subvolume cinder-replicate-5
>[2014-02-12 11:09:25.803093] I [dht-shared.c:311:dht_init_regex] 0-cinder-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-02-12 11:09:26.069135] I [socket.c:3505:socket_init] 0-cinder-client-11: SSL support is NOT enabled
>[2014-02-12 11:09:26.069202] I [socket.c:3520:socket_init] 0-cinder-client-11: using system polling thread
>[2014-02-12 11:09:26.071079] I [socket.c:3505:socket_init] 0-cinder-client-10: SSL support is NOT enabled
>[2014-02-12 11:09:26.071101] I [socket.c:3520:socket_init] 0-cinder-client-10: using system polling thread
>[2014-02-12 11:09:26.071688] I [socket.c:3505:socket_init] 0-cinder-client-9: SSL support is NOT enabled
>[2014-02-12 11:09:26.071709] I [socket.c:3520:socket_init] 0-cinder-client-9: using system polling thread
>[2014-02-12 11:09:26.072339] I [socket.c:3505:socket_init] 0-cinder-client-8: SSL support is NOT enabled
>[2014-02-12 11:09:26.072356] I [socket.c:3520:socket_init] 0-cinder-client-8: using system polling thread
>[2014-02-12 11:09:26.074652] I [socket.c:3505:socket_init] 0-cinder-client-7: SSL support is NOT enabled
>[2014-02-12 11:09:26.074675] I [socket.c:3520:socket_init] 0-cinder-client-7: using system polling thread
>[2014-02-12 11:09:26.079209] I [socket.c:3505:socket_init] 0-cinder-client-6: SSL support is NOT enabled
>[2014-02-12 11:09:26.079233] I [socket.c:3520:socket_init] 0-cinder-client-6: using system polling thread
>[2014-02-12 11:09:26.081252] I [socket.c:3505:socket_init] 0-cinder-client-5: SSL support is NOT enabled
>[2014-02-12 11:09:26.081271] I [socket.c:3520:socket_init] 0-cinder-client-5: using system polling thread
>[2014-02-12 11:09:26.084917] I [socket.c:3505:socket_init] 0-cinder-client-4: SSL support is NOT enabled
>[2014-02-12 11:09:26.084936] I [socket.c:3520:socket_init] 0-cinder-client-4: using system polling thread
>[2014-02-12 11:09:26.088801] I [socket.c:3505:socket_init] 0-cinder-client-3: SSL support is NOT enabled
>[2014-02-12 11:09:26.088841] I [socket.c:3520:socket_init] 0-cinder-client-3: using system polling thread
>[2014-02-12 11:09:26.092415] I [socket.c:3505:socket_init] 0-cinder-client-2: SSL support is NOT enabled
>[2014-02-12 11:09:26.092435] I [socket.c:3520:socket_init] 0-cinder-client-2: using system polling thread
>[2014-02-12 11:09:26.096073] I [socket.c:3505:socket_init] 0-cinder-client-1: SSL support is NOT enabled
>[2014-02-12 11:09:26.096092] I [socket.c:3520:socket_init] 0-cinder-client-1: using system polling thread
>[2014-02-12 11:09:26.099918] I [socket.c:3505:socket_init] 0-cinder-client-0: SSL support is NOT enabled
>[2014-02-12 11:09:26.099938] I [socket.c:3520:socket_init] 0-cinder-client-0: using system polling thread
>[2014-02-12 11:09:26.100009] I [client.c:2171:notify] 0-cinder-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.103206] I [client.c:2171:notify] 0-cinder-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.106203] I [client.c:2171:notify] 0-cinder-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.109645] I [client.c:2171:notify] 0-cinder-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.112724] I [client.c:2171:notify] 0-cinder-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.115773] I [client.c:2171:notify] 0-cinder-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.118841] I [client.c:2171:notify] 0-cinder-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.212155] I [client.c:2171:notify] 0-cinder-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.215296] I [client.c:2171:notify] 0-cinder-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.218358] I [client.c:2171:notify] 0-cinder-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.324953] I [client.c:2171:notify] 0-cinder-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 11:09:26.329284] I [client.c:2171:notify] 0-cinder-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume cinder-client-0
>  2:     type protocol/client
>  3:     option remote-host 10.70.37.96
>  4:     option remote-subvolume /rhs/brick1/c3
>  5:     option transport-type socket
>  6:     option username b769575f-45ec-44d4-a064-b88c317db838
>  7:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>  8:     option filter-O_DIRECT enable
>  9: end-volume
> 10: 
> 11: volume cinder-client-1
> 12:     type protocol/client
> 13:     option remote-host 10.70.37.77
> 14:     option remote-subvolume /rhs/brick1/c4
> 15:     option transport-type socket
> 16:     option username b769575f-45ec-44d4-a064-b88c317db838
> 17:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 18:     option filter-O_DIRECT enable
> 19: end-volume
> 20: 
> 21: volume cinder-replicate-0
> 22:     type cluster/replicate
> 23:     option data-self-heal off
> 24:     option metadata-self-heal off
> 25:     option entry-self-heal off
> 26:     option readdir-failover off
> 27:     option eager-lock enable
> 28:     subvolumes cinder-client-0 cinder-client-1
> 29: end-volume
> 30: 
> 31: volume cinder-client-2
> 32:     type protocol/client
> 33:     option remote-host 10.70.37.121
> 34:     option remote-subvolume /rhs/brick1/c5
> 35:     option transport-type socket
> 36:     option username b769575f-45ec-44d4-a064-b88c317db838
> 37:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 38:     option filter-O_DIRECT enable
> 39: end-volume
> 40: 
> 41: volume cinder-client-3
> 42:     type protocol/client
> 43:     option remote-host 10.70.37.140
> 44:     option remote-subvolume /rhs/brick1/c6
> 45:     option transport-type socket
> 46:     option username b769575f-45ec-44d4-a064-b88c317db838
> 47:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 48:     option filter-O_DIRECT enable
> 49: end-volume
> 50: 
> 51: volume cinder-replicate-1
> 52:     type cluster/replicate
> 53:     option data-self-heal off
> 54:     option metadata-self-heal off
> 55:     option entry-self-heal off
> 56:     option readdir-failover off
> 57:     option eager-lock enable
> 58:     subvolumes cinder-client-2 cinder-client-3
> 59: end-volume
> 60: 
> 61: volume cinder-client-4
> 62:     type protocol/client
> 63:     option remote-host 10.70.37.121
> 64:     option remote-subvolume /rhs/brick1/c9
> 65:     option transport-type socket
> 66:     option username b769575f-45ec-44d4-a064-b88c317db838
> 67:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 68:     option filter-O_DIRECT enable
> 69: end-volume
> 70: 
> 71: volume cinder-client-5
> 72:     type protocol/client
> 73:     option remote-host 10.70.37.140
> 74:     option remote-subvolume /rhs/brick1/c10
> 75:     option transport-type socket
> 76:     option username b769575f-45ec-44d4-a064-b88c317db838
> 77:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 78:     option filter-O_DIRECT enable
> 79: end-volume
> 80: 
> 81: volume cinder-replicate-2
> 82:     type cluster/replicate
> 83:     option data-self-heal off
> 84:     option metadata-self-heal off
> 85:     option entry-self-heal off
> 86:     option readdir-failover off
> 87:     option eager-lock enable
> 88:     subvolumes cinder-client-4 cinder-client-5
> 89: end-volume
> 90: 
> 91: volume cinder-client-6
> 92:     type protocol/client
> 93:     option remote-host 10.70.37.121
> 94:     option remote-subvolume /rhs/brick1/c1
> 95:     option transport-type socket
> 96:     option username b769575f-45ec-44d4-a064-b88c317db838
> 97:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
> 98:     option filter-O_DIRECT enable
> 99: end-volume
>100: 
>101: volume cinder-client-7
>102:     type protocol/client
>103:     option remote-host 10.70.37.140
>104:     option remote-subvolume /rhs/brick1/c2
>105:     option transport-type socket
>106:     option username b769575f-45ec-44d4-a064-b88c317db838
>107:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>108:     option filter-O_DIRECT enable
>109: end-volume
>110: 
>111: volume cinder-replicate-3
>112:     type cluster/replicate
>113:     option data-self-heal off
>114:     option metadata-self-heal off
>115:     option entry-self-heal off
>116:     option readdir-failover off
>117:     option eager-lock enable
>118:     subvolumes cinder-client-6 cinder-client-7
>119: end-volume
>120: 
>121: volume cinder-client-8
>122:     type protocol/client
>123:     option remote-host 10.70.37.96
>124:     option remote-subvolume /rhs/brick1/c11
>125:     option transport-type socket
>126:     option username b769575f-45ec-44d4-a064-b88c317db838
>127:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>128:     option filter-O_DIRECT enable
>129: end-volume
>130: 
>131: volume cinder-client-9
>132:     type protocol/client
>133:     option remote-host 10.70.37.77
>134:     option remote-subvolume /rhs/brick1/c12
>135:     option transport-type socket
>136:     option username b769575f-45ec-44d4-a064-b88c317db838
>137:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>138:     option filter-O_DIRECT enable
>139: end-volume
>140: 
>141: volume cinder-replicate-4
>142:     type cluster/replicate
>143:     option data-self-heal off
>144:     option metadata-self-heal off
>145:     option entry-self-heal off
>146:     option readdir-failover off
>147:     option eager-lock enable
>148:     subvolumes cinder-client-8 cinder-client-9
>149: end-volume
>150: 
>151: volume cinder-client-10
>152:     type protocol/client
>153:     option remote-host 10.70.37.96
>154:     option remote-subvolume /rhs/brick1/c7
>155:     option transport-type socket
>156:     option username b769575f-45ec-44d4-a064-b88c317db838
>157:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>158:     option filter-O_DIRECT enable
>159: end-volume
>160: 
>161: volume cinder-client-11
>162:     type protocol/client
>163:     option remote-host 10.70.37.77
>164:     option remote-subvolume /rhs/brick1/c8
>165:     option transport-type socket
>166:     option username b769575f-45ec-44d4-a064-b88c317db838
>167:     option password 897f4dbc-f9fb-48af-91a9-89c24bbcd548
>168:     option filter-O_DIRECT enable
>169: end-volume
>170: 
>171: volume cinder-replicate-5
>172:     type cluster/replicate
>173:     option data-self-heal off
>174:     option metadata-self-heal off
>175:     option entry-self-heal off
>176:     option readdir-failover off
>177:     option eager-lock enable
>178:     subvolumes cinder-client-10 cinder-client-11
>179: end-volume
>180: 
>181: volume cinder-dht
>182:     type cluster/distribute
>183:     option use-readdirp yes
>184:     option lookup-unhashed yes
>185:     option assert-no-child-down yes
>186:     option readdir-optimize on
>187:     option rebalance-cmd 5
>188:     option node-uuid 304c4df7-e2a3-4d55-aec8-dc645694903e
>189:     option decommissioned-bricks cinder-replicate-5
>190:     subvolumes cinder-replicate-0 cinder-replicate-1 cinder-replicate-2 cinder-replicate-3 cinder-replicate-4 cinder-replicate-5
>191: end-volume
>192: 
>193: volume cinder-write-behind
>194:     type performance/write-behind
>195:     subvolumes cinder-dht
>196: end-volume
>197: 
>198: volume cinder
>199:     type debug/io-stats
>200:     option latency-measurement off
>201:     option count-fop-hits off
>202:     subvolumes cinder-write-behind
>203: end-volume
>204: 
>+------------------------------------------------------------------------------+
>[2014-02-12 11:09:26.369859] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-1: changing port to 49152 (from 0)
>[2014-02-12 11:09:26.373566] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-2: changing port to 49153 (from 0)
>[2014-02-12 11:09:26.373622] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-4: changing port to 49154 (from 0)
>[2014-02-12 11:09:26.373679] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-6: changing port to 49158 (from 0)
>[2014-02-12 11:09:26.373734] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-3: changing port to 49153 (from 0)
>[2014-02-12 11:09:26.423360] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-5: changing port to 49154 (from 0)
>[2014-02-12 11:09:26.423446] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-9: changing port to 49158 (from 0)
>[2014-02-12 11:09:26.423508] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-7: changing port to 49158 (from 0)
>[2014-02-12 11:09:26.423567] I [rpc-clnt.c:1690:rpc_clnt_reconfig] 0-cinder-client-11: changing port to 49159 (from 0)
>[2014-02-12 11:09:26.530346] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 11:09:26.530540] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 11:09:26.530800] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.530998] I [client.c:2103:client_rpc_notify] 0-cinder-client-0: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-12 11:09:26.531048] I [client.c:2103:client_rpc_notify] 0-cinder-client-8: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-12 11:09:26.531483] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.531577] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.531750] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.531893] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.532190] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-2: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.532218] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-2: Connected to 10.70.37.121:49153, attached to remote volume '/rhs/brick1/c5'.
>[2014-02-12 11:09:26.532237] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.532325] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-1: Subvolume 'cinder-client-2' came back up; going online.
>[2014-02-12 11:09:26.532383] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.532503] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.532638] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.532772] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-6: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.532789] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-6: Connected to 10.70.37.121:49158, attached to remote volume '/rhs/brick1/c1'.
>[2014-02-12 11:09:26.532802] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.532918] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-3: Subvolume 'cinder-client-6' came back up; going online.
>[2014-02-12 11:09:26.532964] I [client-handshake.c:1695:select_server_supported_programs] 0-cinder-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 11:09:26.533061] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-4: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.533083] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-4: Connected to 10.70.37.121:49154, attached to remote volume '/rhs/brick1/c9'.
>[2014-02-12 11:09:26.533101] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.533203] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-2: Subvolume 'cinder-client-4' came back up; going online.
>[2014-02-12 11:09:26.533318] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-1: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.533341] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-1: Connected to 10.70.37.77:49152, attached to remote volume '/rhs/brick1/c4'.
>[2014-02-12 11:09:26.533358] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.533452] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-0: Subvolume 'cinder-client-1' came back up; going online.
>[2014-02-12 11:09:26.533673] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-7: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.533701] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-7: Connected to 10.70.37.140:49158, attached to remote volume '/rhs/brick1/c2'.
>[2014-02-12 11:09:26.533719] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.589126] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-2: Server lk version = 1
>[2014-02-12 11:09:26.589213] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-4: Server lk version = 1
>[2014-02-12 11:09:26.589253] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-1: Server lk version = 1
>[2014-02-12 11:09:26.589288] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-7: Server lk version = 1
>[2014-02-12 11:09:26.589361] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-5: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.589440] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-5: Connected to 10.70.37.140:49154, attached to remote volume '/rhs/brick1/c10'.
>[2014-02-12 11:09:26.589459] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.589709] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-6: Server lk version = 1
>[2014-02-12 11:09:26.589860] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-11: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.589888] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-11: Connected to 10.70.37.77:49159, attached to remote volume '/rhs/brick1/c8'.
>[2014-02-12 11:09:26.589905] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.589990] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-5: Subvolume 'cinder-client-11' came back up; going online.
>[2014-02-12 11:09:26.590103] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-9: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.590120] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-9: Connected to 10.70.37.77:49158, attached to remote volume '/rhs/brick1/c12'.
>[2014-02-12 11:09:26.590129] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.590201] I [afr-common.c:3988:afr_notify] 0-cinder-replicate-4: Subvolume 'cinder-client-9' came back up; going online.
>[2014-02-12 11:09:26.590377] I [client-handshake.c:1466:client_setvolume_cbk] 0-cinder-client-3: 'server-pkg-version' key not found, handshaked with older client
>[2014-02-12 11:09:26.590403] I [client-handshake.c:1480:client_setvolume_cbk] 0-cinder-client-3: Connected to 10.70.37.140:49153, attached to remote volume '/rhs/brick1/c6'.
>[2014-02-12 11:09:26.590419] I [client-handshake.c:1492:client_setvolume_cbk] 0-cinder-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 11:09:26.590730] E [client-handshake.c:1778:client_query_portmap_cbk] 0-cinder-client-10: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 11:09:26.591000] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-5: Server lk version = 1
>[2014-02-12 11:09:26.591059] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-11: Server lk version = 1
>[2014-02-12 11:09:26.591091] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-9: Server lk version = 1
>[2014-02-12 11:09:26.591178] I [client.c:2103:client_rpc_notify] 0-cinder-client-10: disconnected from 10.70.37.96:24007. Client process will keep trying to connect to glusterd until brick's port is available. 
>[2014-02-12 11:09:26.624130] I [client-handshake.c:450:client_set_lk_version_cbk] 0-cinder-client-3: Server lk version = 1
>[2014-02-12 11:09:26.777740] I [dht-common.c:2646:dht_setxattr] 0-cinder-dht: fixing the layout of /
>[2014-02-12 11:09:26.853619] I [dht-rebalance.c:1121:gf_defrag_migrate_data] 0-cinder-dht: migrate data called on /
>[2014-02-12 11:09:26.908676] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-0: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-12 11:09:26.993000] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-4: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-12 11:09:27.059002] W [afr-inode-read.c:846:afr_getxattr_node_uuid_cbk] 0-cinder-replicate-5: op_ret (-1): Re-querying afr-child (1/2)
>[2014-02-12 11:09:27.069902] I [dht-rebalance.c:1345:gf_defrag_migrate_data] 0-cinder-dht: Migration operation on dir / took 0.22 secs
>[2014-02-12 11:09:27.197397] I [dht-rebalance.c:1783:gf_defrag_status_get] 0-glusterfs: Rebalance is completed. Time taken is 1.00 secs
>[2014-02-12 11:09:27.197429] I [dht-rebalance.c:1786:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 6, failures: 0, skipped: 0
>[2014-02-12 11:09:27.197869] W [glusterfsd.c:1099:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37438e894d] (-->/lib64/libpthread.so.0() [0x3743c07851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052fd]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 1064321: 862265 | 862266