Attachment 915848 Details for Bug 1064108 – Comment

Comment

comment_6808652.txt (text/plain), 65.94 KB, created by jiademing.dd on 2014-02-12 03:21:23 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: jiademing.dd

Created: 2014-02-12 03:21:23 UTC

Size: 65.94 KB

patch

obsolete

>Description of problem:
>
>    I create a DHT volume named ivcs,then create and write files by postmark,When the postmark is runing, I add-brick, add-brick successful, but lead to some files create failed.
>
>Version-Release number of selected component (if applicable):
>
>
>How reproducible:
>
>
>Steps to Reproduce:
>1.gluster volume create ivcs  node-1:/sdb node-2:/sdb node-1:/sdc node-2:/sdc node-1:/sdd node-2:/sdd node-1:/sde node-2:/sde node-1:/sdf node-2:/sdf node-1:/sdg node-2:/sdg node-1:/sdh node-2:/sdh force
>
>2.run postmark
>set size 2048000 2048000
>set number 50000
>set subdirectories 10
>set location /cluster2/ivcs/
>set read 65536
>set write 65536
>set bias create -1
>set bias read 0
>set transactions 1
>
>3.When postmark running, gluster volume add-brick ivcs node-1:/sdf node-2:/sdf node-1:/sdg node-2:/sdg node-1:/sdh node-2:/sdh force
>
>Actual results:
>
>After add-brick successful, some files create failed by postmark. postmakr can work countinue to create other files.
>[root@node-1 postmark-1.53]# ./postmark 
>PostMark v1.51 : 8/14/01
>pm>load conf
>Reading configuration from file 'conf'
>pm>run
>Creating subdirectories...Done
>Creating files...Error: cannot open '/cluster2/ivcs//s4/729' for writing
>Error: cannot open '/cluster2/ivcs//s0/730' for writing
>Error: cannot open '/cluster2/ivcs//s1/732' for writing
>Error: cannot open '/cluster2/ivcs//s2/733' for writing
>Error: cannot open '/cluster2/ivcs//s8/734' for writing
>Error: cannot open '/cluster2/ivcs//s9/735' for writing
>Error: cannot open '/cluster2/ivcs//s9/738' for writing
>Error: cannot open '/cluster2/ivcs//s8/739' for writing
>Error: cannot open '/cluster2/ivcs//s5/740' for writing
>Error: cannot open '/cluster2/ivcs//s0/741' for writing
>Error: cannot open '/cluster2/ivcs//s5/742' for writing
>Error: cannot open '/cluster2/ivcs//s9/743' for writing
>Error: cannot open '/cluster2/ivcs//s0/744' for writing
>Error: cannot open '/cluster2/ivcs//s2/745' for writing
>Error: cannot open '/cluster2/ivcs//s0/746' for writing
>Error: cannot open '/cluster2/ivcs//s1/747' for writing
>Error: cannot open '/cluster2/ivcs//s7/749' for writing
>Error: cannot open '/cluster2/ivcs//s2/752' for writing
>
>Expected results:
>After add-brick successful, All files create can OKï¼
>
>Additional info:
>
>[2014-02-12 02:26:42.593843] I [client-handshake.c:450:client_set_lk_version_cbk] 0-ivcs-client-1: Server lk version = 1
>[2014-02-12 02:29:08.657680] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-02-12 02:29:08.667848] I [socket.c:3480:socket_init] 1-ivcs-client-13: SSL support is NOT enabled
>[2014-02-12 02:29:08.667909] I [socket.c:3495:socket_init] 1-ivcs-client-13: using system polling thread
>[2014-02-12 02:29:08.668896] I [socket.c:3480:socket_init] 1-ivcs-client-12: SSL support is NOT enabled
>[2014-02-12 02:29:08.668958] I [socket.c:3495:socket_init] 1-ivcs-client-12: using system polling thread
>[2014-02-12 02:29:08.670252] I [socket.c:3480:socket_init] 1-ivcs-client-11: SSL support is NOT enabled
>[2014-02-12 02:29:08.670310] I [socket.c:3495:socket_init] 1-ivcs-client-11: using system polling thread
>[2014-02-12 02:29:08.671652] I [socket.c:3480:socket_init] 1-ivcs-client-10: SSL support is NOT enabled
>[2014-02-12 02:29:08.671717] I [socket.c:3495:socket_init] 1-ivcs-client-10: using system polling thread
>[2014-02-12 02:29:08.672963] I [socket.c:3480:socket_init] 1-ivcs-client-9: SSL support is NOT enabled
>[2014-02-12 02:29:08.673031] I [socket.c:3495:socket_init] 1-ivcs-client-9: using system polling thread
>[2014-02-12 02:29:08.674240] I [socket.c:3480:socket_init] 1-ivcs-client-8: SSL support is NOT enabled
>[2014-02-12 02:29:08.674341] I [socket.c:3495:socket_init] 1-ivcs-client-8: using system polling thread
>[2014-02-12 02:29:08.675520] I [socket.c:3480:socket_init] 1-ivcs-client-7: SSL support is NOT enabled
>[2014-02-12 02:29:08.675660] I [socket.c:3495:socket_init] 1-ivcs-client-7: using system polling thread
>[2014-02-12 02:29:08.676913] I [socket.c:3480:socket_init] 1-ivcs-client-6: SSL support is NOT enabled
>[2014-02-12 02:29:08.676992] I [socket.c:3495:socket_init] 1-ivcs-client-6: using system polling thread
>[2014-02-12 02:29:08.678184] I [socket.c:3480:socket_init] 1-ivcs-client-5: SSL support is NOT enabled
>[2014-02-12 02:29:08.678251] I [socket.c:3495:socket_init] 1-ivcs-client-5: using system polling thread
>[2014-02-12 02:29:08.679468] I [socket.c:3480:socket_init] 1-ivcs-client-4: SSL support is NOT enabled
>[2014-02-12 02:29:08.679530] I [socket.c:3495:socket_init] 1-ivcs-client-4: using system polling thread
>[2014-02-12 02:29:08.680721] I [socket.c:3480:socket_init] 1-ivcs-client-3: SSL support is NOT enabled
>[2014-02-12 02:29:08.680805] I [socket.c:3495:socket_init] 1-ivcs-client-3: using system polling thread
>[2014-02-12 02:29:08.681900] I [socket.c:3480:socket_init] 1-ivcs-client-2: SSL support is NOT enabled
>[2014-02-12 02:29:08.681963] I [socket.c:3495:socket_init] 1-ivcs-client-2: using system polling thread
>[2014-02-12 02:29:08.683122] I [socket.c:3480:socket_init] 1-ivcs-client-1: SSL support is NOT enabled
>[2014-02-12 02:29:08.683168] I [socket.c:3495:socket_init] 1-ivcs-client-1: using system polling thread
>[2014-02-12 02:29:08.684198] I [socket.c:3480:socket_init] 1-ivcs-client-0: SSL support is NOT enabled
>[2014-02-12 02:29:08.684253] I [socket.c:3495:socket_init] 1-ivcs-client-0: using system polling thread
>[2014-02-12 02:29:08.684307] I [client.c:2154:notify] 1-ivcs-client-0: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.687618] I [client.c:2154:notify] 1-ivcs-client-1: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.691028] I [client.c:2154:notify] 1-ivcs-client-2: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.694196] I [client.c:2154:notify] 1-ivcs-client-3: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.697294] I [client.c:2154:notify] 1-ivcs-client-4: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.700408] I [client.c:2154:notify] 1-ivcs-client-5: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.703597] I [client.c:2154:notify] 1-ivcs-client-6: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.706750] I [client.c:2154:notify] 1-ivcs-client-7: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.710039] I [client.c:2154:notify] 1-ivcs-client-8: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.713500] I [client.c:2154:notify] 1-ivcs-client-9: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.716551] I [client.c:2154:notify] 1-ivcs-client-10: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.719619] I [client.c:2154:notify] 1-ivcs-client-11: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.722626] I [client.c:2154:notify] 1-ivcs-client-12: parent translators are ready, attempting connect on transport
>[2014-02-12 02:29:08.725665] I [client.c:2154:notify] 1-ivcs-client-13: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume ivcs-client-0
>  2:     type protocol/client
>  3:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
>  4:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
>  5:     option transport-type tcp
>  6:     option remote-subvolume /sdb
>  7:     option remote-host node-1
>  8: end-volume
>  9: 
> 10: volume ivcs-client-1
> 11:     type protocol/client
> 12:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 13:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 14:     option transport-type tcp
> 15:     option remote-subvolume /sdb
> 16:     option remote-host node-2
> 17: end-volume
> 18: 
> 19: volume ivcs-client-2
> 20:     type protocol/client
> 21:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 22:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 23:     option transport-type tcp
> 24:     option remote-subvolume /sdc
> 25:     option remote-host node-1
> 26: end-volume
> 27: 
> 28: volume ivcs-client-3
> 29:     type protocol/client
> 30:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 31:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 32:     option transport-type tcp
> 33:     option remote-subvolume /sdc
> 34:     option remote-host node-2
> 35: end-volume
> 36: 
> 37: volume ivcs-client-4
> 38:     type protocol/client
> 39:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 40:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 41:     option transport-type tcp
> 42:     option remote-subvolume /sdd
> 43:     option remote-host node-1
> 44: end-volume
> 45: 
> 46: volume ivcs-client-5
> 47:     type protocol/client
> 48:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 49:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 50:     option transport-type tcp
> 51:     option remote-subvolume /sdd
> 52:     option remote-host node-2
> 53: end-volume
> 54: 
> 55: volume ivcs-client-6
> 56:     type protocol/client
> 57:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 58:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 59:     option transport-type tcp
> 60:     option remote-subvolume /sde
> 61:     option remote-host node-1
> 62: end-volume
> 63: 
> 64: volume ivcs-client-7
> 65:     type protocol/client
> 66:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 67:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 68:     option transport-type tcp
> 69:     option remote-subvolume /sde
> 70:     option remote-host node-2
> 71: end-volume
> 72: 
> 73: volume ivcs-client-8
> 74:     type protocol/client
> 75:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 76:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 77:     option transport-type tcp
> 78:     option remote-subvolume /sdf
> 79:     option remote-host node-1
> 80: end-volume
> 81: 
> 82: volume ivcs-client-9
> 83:     type protocol/client
> 84:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 85:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 86:     option transport-type tcp
> 87:     option remote-subvolume /sdf
> 88:     option remote-host node-2
> 89: end-volume
> 90: 
> 91: volume ivcs-client-10
> 92:     type protocol/client
> 93:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
> 94:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
> 95:     option transport-type tcp
> 96:     option remote-subvolume /sdg
> 97:     option remote-host node-1
> 98: end-volume
> 99: 
>100: volume ivcs-client-11
>101:     type protocol/client
>102:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
>103:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
>104:     option transport-type tcp
>105:     option remote-subvolume /sdg
>106:     option remote-host node-2
>107: end-volume
>108: 
>109: volume ivcs-client-12
>110:     type protocol/client
>111:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
>112:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
>113:     option transport-type tcp
>114:     option remote-subvolume /sdh
>115:     option remote-host node-1
>116: end-volume
>117: 
>118: volume ivcs-client-13
>119:     type protocol/client
>120:     option password 4f57a24d-ce53-40c2-b2ac-3cbce4c9a5d4
>121:     option username 68057ba4-5089-4f71-a684-42a9b0c52a82
>122:     option transport-type tcp
>123:     option remote-subvolume /sdh
>124:     option remote-host node-2
>125: end-volume
>126: 
>127: volume ivcs-dht
>128:     type cluster/distribute
>129:     subvolumes ivcs-client-0 ivcs-client-1 ivcs-client-2 ivcs-client-3 ivcs-client-4 ivcs-client-5 ivcs-client-6 ivcs-client-7 ivcs-client-8 ivcs-client-9 ivcs-client-10 ivcs-client-11 ivcs-client-12 ivcs-client-13
>130: end-volume
>131: 
>132: volume ivcs-write-behind
>133:     type performance/write-behind
>134:     subvolumes ivcs-dht
>135: end-volume
>136: 
>137: volume ivcs-read-ahead
>138:     type performance/read-ahead
>139:     subvolumes ivcs-write-behind
>140: end-volume
>141: 
>142: volume ivcs-io-cache
>143:     type performance/io-cache
>144:     subvolumes ivcs-read-ahead
>145: end-volume
>146: 
>147: volume ivcs-quick-read
>148:     type performance/quick-read
>149:     subvolumes ivcs-io-cache
>150: end-volume
>151: 
>152: volume ivcs-open-behind
>153:     type performance/open-behind
>154:     subvolumes ivcs-quick-read
>155: end-volume
>156: 
>157: volume ivcs-md-cache
>158:     type performance/md-cache
>159:     subvolumes ivcs-open-behind
>160: end-volume
>161: 
>162: volume ivcs
>163:     type debug/io-stats
>164:     option count-fop-hits off
>165:     option latency-measurement off
>166:     subvolumes ivcs-md-cache
>167: end-volume
>
>+------------------------------------------------------------------------------+
>[2014-02-12 02:29:08.730004] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-1: changing port to 49152 (from 0)
>[2014-02-12 02:29:08.730070] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-3: changing port to 49153 (from 0)
>[2014-02-12 02:29:08.730273] W [socket.c:514:__socket_rwv] 1-ivcs-client-1: readv failed (No data available)
>[2014-02-12 02:29:08.734417] W [socket.c:514:__socket_rwv] 1-ivcs-client-3: readv failed (No data available)
>[2014-02-12 02:29:08.738016] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-0: changing port to 49152 (from 0)
>[2014-02-12 02:29:08.738092] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-2: changing port to 49153 (from 0)
>[2014-02-12 02:29:08.738148] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-4: changing port to 49154 (from 0)
>[2014-02-12 02:29:08.738180] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-6: changing port to 49155 (from 0)
>[2014-02-12 02:29:08.738209] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738246] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-10: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738280] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-12: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738312] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-5: changing port to 49154 (from 0)
>[2014-02-12 02:29:08.738341] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-7: changing port to 49155 (from 0)
>[2014-02-12 02:29:08.738363] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738419] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-11: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738444] E [client-handshake.c:1742:client_query_portmap_cbk] 1-ivcs-client-13: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-02-12 02:29:08.738703] W [socket.c:514:__socket_rwv] 1-ivcs-client-0: readv failed (No data available)
>[2014-02-12 02:29:08.742135] W [socket.c:514:__socket_rwv] 1-ivcs-client-2: readv failed (No data available)
>[2014-02-12 02:29:08.745300] W [socket.c:514:__socket_rwv] 1-ivcs-client-4: readv failed (No data available)
>[2014-02-12 02:29:08.748382] W [socket.c:514:__socket_rwv] 1-ivcs-client-6: readv failed (No data available)
>[2014-02-12 02:29:08.751564] W [socket.c:514:__socket_rwv] 1-ivcs-client-8: readv failed (No data available)
>[2014-02-12 02:29:08.751682] I [client.c:2097:client_rpc_notify] 1-ivcs-client-8: disconnected
>[2014-02-12 02:29:08.751701] W [socket.c:514:__socket_rwv] 1-ivcs-client-10: readv failed (No data available)
>[2014-02-12 02:29:08.751719] I [client.c:2097:client_rpc_notify] 1-ivcs-client-10: disconnected
>[2014-02-12 02:29:08.751735] W [socket.c:514:__socket_rwv] 1-ivcs-client-12: readv failed (No data available)
>[2014-02-12 02:29:08.751752] I [client.c:2097:client_rpc_notify] 1-ivcs-client-12: disconnected
>[2014-02-12 02:29:08.751765] W [socket.c:514:__socket_rwv] 1-ivcs-client-5: readv failed (No data available)
>[2014-02-12 02:29:08.755012] W [socket.c:514:__socket_rwv] 1-ivcs-client-7: readv failed (No data available)
>[2014-02-12 02:29:08.758159] W [socket.c:514:__socket_rwv] 1-ivcs-client-9: readv failed (No data available)
>[2014-02-12 02:29:08.758221] I [client.c:2097:client_rpc_notify] 1-ivcs-client-9: disconnected
>[2014-02-12 02:29:08.758239] W [socket.c:514:__socket_rwv] 1-ivcs-client-11: readv failed (No data available)
>[2014-02-12 02:29:08.758257] I [client.c:2097:client_rpc_notify] 1-ivcs-client-11: disconnected
>[2014-02-12 02:29:08.758270] W [socket.c:514:__socket_rwv] 1-ivcs-client-13: readv failed (No data available)
>[2014-02-12 02:29:08.758306] I [client.c:2097:client_rpc_notify] 1-ivcs-client-13: disconnected
>[2014-02-12 02:29:08.758512] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.759115] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.759275] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-1: Connected to 10.10.15.228:49152, attached to remote volume '/sdb'.
>[2014-02-12 02:29:08.759304] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.759408] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.759892] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.760330] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.760732] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.761183] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-3: Connected to 10.10.15.228:49153, attached to remote volume '/sdc'.
>[2014-02-12 02:29:08.761214] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.761317] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-1: Server lk version = 1
>[2014-02-12 02:29:08.761358] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-0: Connected to 10.10.15.226:49152, attached to remote volume '/sdb'.
>[2014-02-12 02:29:08.761370] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.761746] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-2: Connected to 10.10.15.226:49153, attached to remote volume '/sdc'.
>[2014-02-12 02:29:08.761777] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.762262] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-4: Connected to 10.10.15.226:49154, attached to remote volume '/sdd'.
>[2014-02-12 02:29:08.762321] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.762684] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.762814] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:08.762960] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-3: Server lk version = 1
>[2014-02-12 02:29:08.763008] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-0: Server lk version = 1
>[2014-02-12 02:29:08.763069] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-2: Server lk version = 1
>[2014-02-12 02:29:08.763091] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-4: Server lk version = 1
>[2014-02-12 02:29:08.763125] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-6: Connected to 10.10.15.226:49155, attached to remote volume '/sde'.
>[2014-02-12 02:29:08.763138] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.763725] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-6: Server lk version = 1
>[2014-02-12 02:29:08.763884] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-5: Connected to 10.10.15.228:49154, attached to remote volume '/sdd'.
>[2014-02-12 02:29:08.763940] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.764414] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-7: Connected to 10.10.15.228:49155, attached to remote volume '/sde'.
>[2014-02-12 02:29:08.764436] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:08.771482] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 1
>[2014-02-12 02:29:08.771663] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-7: Server lk version = 1
>[2014-02-12 02:29:08.771737] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-5: Server lk version = 1
>[2014-02-12 02:29:08.774136] I [client.c:2163:notify] 0-ivcs-client-0: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774213] I [client.c:2163:notify] 0-ivcs-client-1: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774237] I [client.c:2163:notify] 0-ivcs-client-2: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774262] I [client.c:2163:notify] 0-ivcs-client-3: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774288] I [client.c:2163:notify] 0-ivcs-client-4: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774311] I [client.c:2163:notify] 0-ivcs-client-5: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774328] I [client.c:2163:notify] 0-ivcs-client-6: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.774353] I [client.c:2163:notify] 0-ivcs-client-7: current graph is no longer active, destroying rpc_client 
>[2014-02-12 02:29:08.775405] W [socket.c:514:__socket_rwv] 0-ivcs-client-0: readv failed (No data available)
>[2014-02-12 02:29:08.775458] I [client.c:2097:client_rpc_notify] 0-ivcs-client-0: disconnected
>[2014-02-12 02:29:08.775481] W [socket.c:514:__socket_rwv] 0-ivcs-client-1: readv failed (No data available)
>[2014-02-12 02:29:08.775499] I [client.c:2097:client_rpc_notify] 0-ivcs-client-1: disconnected
>[2014-02-12 02:29:08.775513] W [socket.c:514:__socket_rwv] 0-ivcs-client-2: readv failed (No data available)
>[2014-02-12 02:29:08.775529] I [client.c:2097:client_rpc_notify] 0-ivcs-client-2: disconnected
>[2014-02-12 02:29:08.775564] W [socket.c:514:__socket_rwv] 0-ivcs-client-3: readv failed (No data available)
>[2014-02-12 02:29:08.775598] I [client.c:2097:client_rpc_notify] 0-ivcs-client-3: disconnected
>[2014-02-12 02:29:08.775617] W [socket.c:514:__socket_rwv] 0-ivcs-client-4: readv failed (No data available)
>[2014-02-12 02:29:08.775635] I [client.c:2097:client_rpc_notify] 0-ivcs-client-4: disconnected
>[2014-02-12 02:29:08.775649] W [socket.c:514:__socket_rwv] 0-ivcs-client-5: readv failed (No data available)
>[2014-02-12 02:29:08.775666] I [client.c:2097:client_rpc_notify] 0-ivcs-client-5: disconnected
>[2014-02-12 02:29:08.775680] W [socket.c:514:__socket_rwv] 0-ivcs-client-6: readv failed (No data available)
>[2014-02-12 02:29:08.775695] I [client.c:2097:client_rpc_notify] 0-ivcs-client-6: disconnected
>[2014-02-12 02:29:08.775709] W [socket.c:514:__socket_rwv] 0-ivcs-client-7: readv failed (No data available)
>[2014-02-12 02:29:08.775726] I [client.c:2097:client_rpc_notify] 0-ivcs-client-7: disconnected
>[2014-02-12 02:29:08.782938] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-8
>[2014-02-12 02:29:08.782981] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:08.782995] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-10
>[2014-02-12 02:29:08.783020] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:08.783030] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-12
>[2014-02-12 02:29:08.783039] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:08.812604] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:2d96b506-9a4f-424c-b1ae-455405092dab>. holes=4 overlaps=1
>[2014-02-12 02:29:08.812676] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 2d96b506-9a4f-424c-b1ae-455405092dab: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.812693] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 26979: GETATTR 140200516260664 (2d96b506-9a4f-424c-b1ae-455405092dab) resolution failed
>[2014-02-12 02:29:08.814137] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:45b126ad-f8eb-48be-82aa-93b8937f7dbf>. holes=3 overlaps=0
>[2014-02-12 02:29:08.814204] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 45b126ad-f8eb-48be-82aa-93b8937f7dbf: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.814218] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 26980: GETATTR 140200516260040 (45b126ad-f8eb-48be-82aa-93b8937f7dbf) resolution failed
>[2014-02-12 02:29:08.844900] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:880395e8-2735-4996-b41b-35f61790b3b7>. holes=2 overlaps=1
>[2014-02-12 02:29:08.844967] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 880395e8-2735-4996-b41b-35f61790b3b7: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.844982] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27018: GETATTR 140200516260352 (880395e8-2735-4996-b41b-35f61790b3b7) resolution failed
>[2014-02-12 02:29:08.846928] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:6660932b-9d64-496a-b698-31cd1541c0a6>. holes=5 overlaps=1
>[2014-02-12 02:29:08.846988] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 6660932b-9d64-496a-b698-31cd1541c0a6: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.847004] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27019: GETATTR 140200516260196 (6660932b-9d64-496a-b698-31cd1541c0a6) resolution failed
>[2014-02-12 02:29:08.849989] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:f69576f7-48a8-4a24-9427-e6ba9800ba4c>. holes=3 overlaps=0
>[2014-02-12 02:29:08.850055] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: f69576f7-48a8-4a24-9427-e6ba9800ba4c: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.850069] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27020: GETATTR 140200516261444 (f69576f7-48a8-4a24-9427-e6ba9800ba4c) resolution failed
>[2014-02-12 02:29:08.852021] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:66e99490-3dab-4fd1-b5b5-e04064d698b1>. holes=5 overlaps=0
>[2014-02-12 02:29:08.852088] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 66e99490-3dab-4fd1-b5b5-e04064d698b1: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.852104] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27021: GETATTR 140200516261600 (66e99490-3dab-4fd1-b5b5-e04064d698b1) resolution failed
>[2014-02-12 02:29:08.905428] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:66e99490-3dab-4fd1-b5b5-e04064d698b1>. holes=7 overlaps=0
>[2014-02-12 02:29:08.905487] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 66e99490-3dab-4fd1-b5b5-e04064d698b1: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.905500] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27096: GETATTR 140200516261600 (66e99490-3dab-4fd1-b5b5-e04064d698b1) resolution failed
>[2014-02-12 02:29:08.906547] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:f69576f7-48a8-4a24-9427-e6ba9800ba4c>. holes=5 overlaps=0
>[2014-02-12 02:29:08.906636] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: f69576f7-48a8-4a24-9427-e6ba9800ba4c: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.906663] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27097: GETATTR 140200516261444 (f69576f7-48a8-4a24-9427-e6ba9800ba4c) resolution failed
>[2014-02-12 02:29:08.907791] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:b43d826d-7da4-4fbd-8098-7e8988cd916e>. holes=4 overlaps=1
>[2014-02-12 02:29:08.907846] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: b43d826d-7da4-4fbd-8098-7e8988cd916e: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.907859] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27098: GETATTR 140200516260820 (b43d826d-7da4-4fbd-8098-7e8988cd916e) resolution failed
>[2014-02-12 02:29:08.908869] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:45b126ad-f8eb-48be-82aa-93b8937f7dbf>. holes=5 overlaps=0
>[2014-02-12 02:29:08.908924] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 45b126ad-f8eb-48be-82aa-93b8937f7dbf: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.908938] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27099: GETATTR 140200516260040 (45b126ad-f8eb-48be-82aa-93b8937f7dbf) resolution failed
>[2014-02-12 02:29:08.910008] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:b43d826d-7da4-4fbd-8098-7e8988cd916e>. holes=2 overlaps=1
>[2014-02-12 02:29:08.910064] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: b43d826d-7da4-4fbd-8098-7e8988cd916e: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.910076] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27100: GETATTR 140200516260820 (b43d826d-7da4-4fbd-8098-7e8988cd916e) resolution failed
>[2014-02-12 02:29:08.911110] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:66e99490-3dab-4fd1-b5b5-e04064d698b1>. holes=7 overlaps=0
>[2014-02-12 02:29:08.911165] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 66e99490-3dab-4fd1-b5b5-e04064d698b1: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.911178] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27101: GETATTR 140200516261600 (66e99490-3dab-4fd1-b5b5-e04064d698b1) resolution failed
>[2014-02-12 02:29:08.912257] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:45b126ad-f8eb-48be-82aa-93b8937f7dbf>. holes=3 overlaps=0
>[2014-02-12 02:29:08.912311] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 45b126ad-f8eb-48be-82aa-93b8937f7dbf: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.912323] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27102: GETATTR 140200516260040 (45b126ad-f8eb-48be-82aa-93b8937f7dbf) resolution failed
>[2014-02-12 02:29:08.913448] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:6660932b-9d64-496a-b698-31cd1541c0a6>. holes=5 overlaps=1
>[2014-02-12 02:29:08.913500] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 6660932b-9d64-496a-b698-31cd1541c0a6: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.913512] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27103: GETATTR 140200516260196 (6660932b-9d64-496a-b698-31cd1541c0a6) resolution failed
>[2014-02-12 02:29:08.914614] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:45b126ad-f8eb-48be-82aa-93b8937f7dbf>. holes=4 overlaps=1
>[2014-02-12 02:29:08.914667] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 45b126ad-f8eb-48be-82aa-93b8937f7dbf: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.914680] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27104: GETATTR 140200516260040 (45b126ad-f8eb-48be-82aa-93b8937f7dbf) resolution failed
>[2014-02-12 02:29:08.915665] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:880395e8-2735-4996-b41b-35f61790b3b7>. holes=4 overlaps=1
>[2014-02-12 02:29:08.915720] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 880395e8-2735-4996-b41b-35f61790b3b7: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.915744] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27105: GETATTR 140200516260352 (880395e8-2735-4996-b41b-35f61790b3b7) resolution failed
>[2014-02-12 02:29:08.949429] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:d125d9d1-22b2-4723-86b2-111f38101e69>. holes=5 overlaps=2
>[2014-02-12 02:29:08.949487] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: d125d9d1-22b2-4723-86b2-111f38101e69: failed to resolve (Invalid argument)
>[2014-02-12 02:29:08.949500] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27214: GETATTR 140200516261132 (d125d9d1-22b2-4723-86b2-111f38101e69) resolution failed
>[2014-02-12 02:29:09.010783] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in <gfid:6660932b-9d64-496a-b698-31cd1541c0a6>. holes=5 overlaps=1
>[2014-02-12 02:29:09.498606] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk] 0-fuse: 6660932b-9d64-496a-b698-31cd1541c0a6: failed to resolve (Invalid argument)
>[2014-02-12 02:29:09.498638] E [fuse-bridge.c:734:fuse_getattr_resume] 0-glusterfs-fuse: 27365: GETATTR 140200516260196 (6660932b-9d64-496a-b698-31cd1541c0a6) resolution failed
>[2014-02-12 02:29:09.513406] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-8
>[2014-02-12 02:29:09.513459] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:09.513471] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-10
>[2014-02-12 02:29:09.513480] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:09.513489] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-12
>[2014-02-12 02:29:09.513497] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:10.004619] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-8
>[2014-02-12 02:29:10.004667] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:10.004677] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-10
>[2014-02-12 02:29:10.004685] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:10.004694] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-12
>[2014-02-12 02:29:10.004702] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:11.013418] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-8
>[2014-02-12 02:29:11.013464] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:11.013475] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-10
>[2014-02-12 02:29:11.013483] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:11.013491] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-12
>[2014-02-12 02:29:11.013499] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:12.019268] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-8
>[2014-02-12 02:29:12.019424] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:12.019447] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-10
>[2014-02-12 02:29:12.019457] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:12.019465] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-12
>[2014-02-12 02:29:12.019474] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:12.538877] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-8: changing port to 49156 (from 0)
>[2014-02-12 02:29:12.538961] W [socket.c:514:__socket_rwv] 1-ivcs-client-8: readv failed (No data available)
>[2014-02-12 02:29:12.545350] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:12.545615] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-8: Connected to 10.10.15.226:49156, attached to remote volume '/sdf'.
>[2014-02-12 02:29:12.545633] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:12.546026] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-8: Server lk version = 1
>[2014-02-12 02:29:12.548840] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-10: changing port to 49157 (from 0)
>[2014-02-12 02:29:12.548931] W [socket.c:514:__socket_rwv] 1-ivcs-client-10: readv failed (No data available)
>[2014-02-12 02:29:12.558392] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:12.562084] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-10: Connected to 10.10.15.226:49157, attached to remote volume '/sdg'.
>[2014-02-12 02:29:12.562638] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:12.562801] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-12: changing port to 49158 (from 0)
>[2014-02-12 02:29:12.563501] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-10: Server lk version = 1
>[2014-02-12 02:29:12.563558] W [socket.c:514:__socket_rwv] 1-ivcs-client-12: readv failed (No data available)
>[2014-02-12 02:29:12.567250] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:12.567784] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-12: Connected to 10.10.15.226:49158, attached to remote volume '/sdh'.
>[2014-02-12 02:29:12.567819] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-12: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:12.568307] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-12: Server lk version = 1
>[2014-02-12 02:29:12.709876] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:12.709982] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:12.710268] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:12.755045] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:12.755102] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:12.755257] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.002767] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-9
>[2014-02-12 02:29:13.003007] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-11
>[2014-02-12 02:29:13.003123] W [dht-diskusage.c:45:dht_du_info_cbk] 1-ivcs-dht: failed to get disk info from ivcs-client-13
>[2014-02-12 02:29:13.020596] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.020640] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.020667] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.062229] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s2. holes=6 overlaps=0
>[2014-02-12 02:29:13.062277] W [dht-selfheal.c:931:dht_selfheal_directory] 1-ivcs-dht: 3 subvolumes down -- not fixing
>[2014-02-12 02:29:13.080984] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.081171] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.081231] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.100051] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.100082] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.100101] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.123231] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s4. holes=2 overlaps=1
>[2014-02-12 02:29:13.123279] W [dht-selfheal.c:931:dht_selfheal_directory] 1-ivcs-dht: 3 subvolumes down -- not fixing
>[2014-02-12 02:29:13.178378] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.178540] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.178757] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:13.278612] W [dht-selfheal.c:931:dht_selfheal_directory] 1-ivcs-dht: 3 subvolumes down -- not fixing
>[2014-02-12 02:29:13.464756] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s8. holes=6 overlaps=2
>[2014-02-12 02:29:13.464805] W [dht-selfheal.c:931:dht_selfheal_directory] 1-ivcs-dht: 3 subvolumes down -- not fixing
>[2014-02-12 02:29:13.654434] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s7. holes=2 overlaps=1
>[2014-02-12 02:29:13.654481] W [dht-selfheal.c:931:dht_selfheal_directory] 1-ivcs-dht: 3 subvolumes down -- not fixing
>[2014-02-12 02:29:13.708411] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:13.708476] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:13.708507] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:13.708519] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:13.708713] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:13.708733] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:13.880788] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-13: changing port to 49158 (from 0)
>[2014-02-12 02:29:13.880861] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-9: changing port to 49156 (from 0)
>[2014-02-12 02:29:13.880886] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 1-ivcs-client-11: changing port to 49157 (from 0)
>[2014-02-12 02:29:13.880922] W [socket.c:514:__socket_rwv] 1-ivcs-client-13: readv failed (No data available)
>[2014-02-12 02:29:13.884172] W [socket.c:514:__socket_rwv] 1-ivcs-client-9: readv failed (No data available)
>[2014-02-12 02:29:13.887417] W [socket.c:514:__socket_rwv] 1-ivcs-client-11: readv failed (No data available)
>[2014-02-12 02:29:13.891979] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:13.892098] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:13.892185] I [client-handshake.c:1659:select_server_supported_programs] 1-ivcs-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-02-12 02:29:13.892455] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-9: Connected to 10.10.15.228:49156, attached to remote volume '/sdf'.
>[2014-02-12 02:29:13.892473] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:13.892642] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-9: Server lk version = 1
>[2014-02-12 02:29:13.895294] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-11: Connected to 10.10.15.228:49157, attached to remote volume '/sdg'.
>[2014-02-12 02:29:13.895343] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:13.895654] I [client-handshake.c:1456:client_setvolume_cbk] 1-ivcs-client-13: Connected to 10.10.15.228:49158, attached to remote volume '/sdh'.
>[2014-02-12 02:29:13.895682] I [client-handshake.c:1468:client_setvolume_cbk] 1-ivcs-client-13: Server and Client lk-version numbers are not same, reopening the fds
>[2014-02-12 02:29:13.896448] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-11: Server lk version = 1
>[2014-02-12 02:29:13.896954] I [client-handshake.c:450:client_set_lk_version_cbk] 1-ivcs-client-13: Server lk version = 1
>[2014-02-12 02:29:14.016224] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.016321] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.016524] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.017060] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.017097] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.017115] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.035700] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.035750] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.036011] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.036537] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.036618] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.036645] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.073019] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s3. holes=4 overlaps=3
>[2014-02-12 02:29:14.423119] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.423359] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.423447] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.424074] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.424130] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.424195] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.445977] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.446256] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.446296] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.446486] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.446507] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.446526] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.465793] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s0. holes=2 overlaps=1
>[2014-02-12 02:29:14.490793] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.490833] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.490858] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.491058] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.491085] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.491150] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:14.530980] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s5. holes=3 overlaps=2
>[2014-02-12 02:29:14.581333] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s6. holes=4 overlaps=1
>[2014-02-12 02:29:14.704164] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s8. holes=4 overlaps=2
>[2014-02-12 02:29:15.009435] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.009474] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.009698] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.009730] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.009933] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.009959] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073383] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073490] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073731] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073761] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073782] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.073803] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.095710] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.095807] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.095836] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.096295] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.096336] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.096356] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.171698] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.172081] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.172119] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.172450] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.172484] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.172594] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.222846] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.222903] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.223442] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.223732] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.223760] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.223777] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:15.746733] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.746778] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:15.746802] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.746813] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:15.746858] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.746870] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:15.746956] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.746971] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:15.746995] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.747005] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:15.747076] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:15.747087] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.002290] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.002753] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.002804] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.003195] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.003228] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.003246] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.026875] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s7. holes=5 overlaps=1
>[2014-02-12 02:29:16.166899] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.167032] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.167285] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.167893] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.167931] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.167950] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293077] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293122] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293165] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293191] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293217] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.293747] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:16.591242] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s9. holes=4 overlaps=0
>[2014-02-12 02:29:16.766233] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766280] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.766306] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766319] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.766477] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766498] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.766518] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766528] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.766547] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766557] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:16.766753] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:16.766768] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.066949] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.066988] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.067044] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.067074] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.067130] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.067492] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:17.157118] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s4. holes=6 overlaps=0
>[2014-02-12 02:29:17.778961] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779022] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.779199] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779225] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.779247] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779274] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.779298] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779307] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.779325] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779334] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.779526] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:17.779542] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:17.942520] I [dht-layout.c:638:dht_layout_normalize] 1-ivcs-dht: found anomalies in /s2. holes=6 overlaps=2
>[2014-02-12 02:29:18.104211] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.104251] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.104285] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.104438] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.112895] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.112924] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory
>[2014-02-12 02:29:18.783225] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783291] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:18.783317] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783328] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:18.783351] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783362] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:18.783486] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783508] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:18.783726] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783740] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:18.783771] I [dht-layout.c:726:dht_layout_dir_mismatch] 1-ivcs-dht: / - disk layout missing
>[2014-02-12 02:29:18.783781] I [dht-common.c:623:dht_revalidate_cbk] 1-ivcs-dht: mismatching layouts for /
>[2014-02-12 02:29:19.103408] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-11: remote operation failed: No such file or directory
>[2014-02-12 02:29:19.103447] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-12: remote operation failed: No such file or directory
>[2014-02-12 02:29:19.103467] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-10: remote operation failed: No such file or directory
>[2014-02-12 02:29:19.103495] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-13: remote operation failed: No such file or directory
>[2014-02-12 02:29:19.103519] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-8: remote operation failed: No such file or directory
>[2014-02-12 02:29:19.103770] W [client-rpc-fops.c:526:client3_3_stat_cbk] 1-ivcs-client-9: remote operation failed: No such file or directory

Actions: View

Attachments on bug 1064108: 915848