Attachment 899401 Details for Bug 1101438 – nfs logs

nfs logs

nfs.log (text/x-log), 98.42 KB, created by Saurabh on 2014-05-27 08:03:09 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Saurabh

Created: 2014-05-27 08:03:09 UTC

Size: 98.42 KB

patch

obsolete

>[2014-05-25 22:40:06.405041] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-25 22:40:06.422615] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
>[2014-05-25 22:40:06.446251] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON
>[2014-05-25 22:40:06.451251] I [nfs.c:1353:init] 0-nfs: NFS service started
>[2014-05-25 22:40:06.451586] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-25 22:40:06.452507] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2014-05-25 22:40:06.452575] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2014-05-25 22:40:06.452625] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2014-05-25 22:40:06.452698] I [client.c:2280:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume testvol-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host 10.70.37.62
>  5:     option remote-subvolume /tmp/brick
>  6:     option transport-type socket
>  7:     option username 6c4a83e9-23e7-47f7-ae92-dec72d4ee6a8
>  8:     option password 4a706d27-89b6-4b6d-87d4-48cc949815c9
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume testvol-dht
> 13:     type cluster/distribute
> 14:     subvolumes testvol-client-0
> 15: end-volume
> 16:  
> 17: volume testvol-write-behind
> 18:     type performance/write-behind
> 19:     subvolumes testvol-dht
> 20: end-volume
> 21:  
> 22: volume testvol
> 23:     type debug/io-stats
> 24:     option latency-measurement off
> 25:     option count-fop-hits off
> 26:     subvolumes testvol-write-behind
> 27: end-volume
> 28:  
> 29: volume nfs-server
> 30:     type nfs/server
> 31:     option rpc-auth.auth-glusterfs on
> 32:     option rpc-auth.auth-unix on
> 33:     option rpc-auth.auth-null on
> 34:     option rpc-auth.ports.insecure on
> 35:     option rpc-auth-allow-insecure on
> 36:     option transport-type socket
> 37:     option transport.socket.listen-port 2049
> 38:     option nfs.dynamic-volumes on
> 39:     option nfs.nlm on
> 40:     option nfs.drc on
> 41:     option rpc-auth.addr.testvol.allow *
> 42:     option nfs3.testvol.volume-id 036848d4-e54d-4606-b21f-bb0e1d380bc0
> 43:     subvolumes testvol
> 44: end-volume
> 45:  
>+------------------------------------------------------------------------------+
>[2014-05-25 22:40:06.457016] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49152 (from 0)
>[2014-05-25 22:40:06.460396] I [client-handshake.c:1415:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-25 22:40:06.463470] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/tmp/brick'.
>[2014-05-25 22:40:06.463514] I [client-handshake.c:1212:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-25 22:40:06.463822] I [client-handshake.c:188:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1
>[2014-05-25 22:40:06.464591] I [dht-layout.c:653:dht_layout_normalize] 0-testvol-dht: found anomalies in /. holes=1 overlaps=0
>[2014-05-25 22:40:39.697322] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:40:39.730476] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:40:39.735279] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-25 22:40:39.735682] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:51:22.735395] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:51:22.759200] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:51:49.193918] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:51:49.220479] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:52:19.510215] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:52:19.533004] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:54:17.474515] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:54:17.511561] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:54:42.459596] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:54:42.484899] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 22:54:51.958674] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:54:51.990855] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 22:54:51.995246] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-25 22:54:51.995546] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 23:03:36.635944] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 23:03:36.668751] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 23:03:43.692258] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 23:03:43.715562] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 23:40:12.355766] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 23:40:12.381863] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-25 23:41:46.429087] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-25 23:41:46.452264] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 18:54:12.571391] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 18:54:12.606083] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 18:54:12.609921] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-26 18:54:12.610198] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 18:54:36.622463] W [socket.c:522:__socket_rwv] 0-testvol-client-0: readv on 10.70.37.62:49152 failed (No data available)
>[2014-05-26 18:54:36.622679] I [client.c:2215:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 18:59:06.283180] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-26 18:59:06.305907] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
>[2014-05-26 18:59:06.373445] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON
>[2014-05-26 18:59:06.379214] I [nfs.c:1353:init] 0-nfs: NFS service started
>[2014-05-26 18:59:06.379490] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-26 18:59:06.390941] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2014-05-26 18:59:06.391073] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2014-05-26 18:59:06.391144] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2014-05-26 18:59:06.391341] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.394994] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.398282] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.401601] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.404687] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.407810] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.415946] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.421056] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.424093] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.427183] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.430162] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport
>[2014-05-26 18:59:06.433157] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dist-rep-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host 10.70.37.62
>  5:     option remote-subvolume /bricks/d1r1
>  6:     option transport-type socket
>  7:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>  8:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume dist-rep-client-1
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host 10.70.37.215
> 16:     option remote-subvolume /bricks/d1r2
> 17:     option transport-type socket
> 18:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 19:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume dist-rep-replicate-0
> 24:     type cluster/replicate
> 25:     subvolumes dist-rep-client-0 dist-rep-client-1
> 26: end-volume
> 27:  
> 28: volume dist-rep-client-2
> 29:     type protocol/client
> 30:     option ping-timeout 42
> 31:     option remote-host 10.70.37.44
> 32:     option remote-subvolume /bricks/d2r1
> 33:     option transport-type socket
> 34:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 35:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 36:     option send-gids true
> 37: end-volume
> 38:  
> 39: volume dist-rep-client-3
> 40:     type protocol/client
> 41:     option ping-timeout 42
> 42:     option remote-host 10.70.37.201
> 43:     option remote-subvolume /bricks/d2r2
> 44:     option transport-type socket
> 45:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 46:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 47:     option send-gids true
> 48: end-volume
> 49:  
> 50: volume dist-rep-replicate-1
> 51:     type cluster/replicate
> 52:     subvolumes dist-rep-client-2 dist-rep-client-3
> 53: end-volume
> 54:  
> 55: volume dist-rep-client-4
> 56:     type protocol/client
> 57:     option ping-timeout 42
> 58:     option remote-host 10.70.37.62
> 59:     option remote-subvolume /bricks/d3r1
> 60:     option transport-type socket
> 61:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 62:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 63:     option send-gids true
> 64: end-volume
> 65:  
> 66: volume dist-rep-client-5
> 67:     type protocol/client
> 68:     option ping-timeout 42
> 69:     option remote-host 10.70.37.215
> 70:     option remote-subvolume /bricks/d3r2
> 71:     option transport-type socket
> 72:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 73:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 74:     option send-gids true
> 75: end-volume
> 76:  
> 77: volume dist-rep-replicate-2
> 78:     type cluster/replicate
> 79:     subvolumes dist-rep-client-4 dist-rep-client-5
> 80: end-volume
> 81:  
> 82: volume dist-rep-client-6
> 83:     type protocol/client
> 84:     option ping-timeout 42
> 85:     option remote-host 10.70.37.44
> 86:     option remote-subvolume /bricks/d4r1
> 87:     option transport-type socket
> 88:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 89:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 90:     option send-gids true
> 91: end-volume
> 92:  
> 93: volume dist-rep-client-7
> 94:     type protocol/client
> 95:     option ping-timeout 42
> 96:     option remote-host 10.70.37.201
> 97:     option remote-subvolume /bricks/d4r2
> 98:     option transport-type socket
> 99:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>100:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>101:     option send-gids true
>102: end-volume
>103:  
>104: volume dist-rep-replicate-3
>105:     type cluster/replicate
>106:     subvolumes dist-rep-client-6 dist-rep-client-7
>107: end-volume
>108:  
>109: volume dist-rep-client-8
>110:     type protocol/client
>111:     option ping-timeout 42
>112:     option remote-host 10.70.37.62
>113:     option remote-subvolume /bricks/d5r1
>114:     option transport-type socket
>115:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>116:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>117:     option send-gids true
>118: end-volume
>119:  
>120: volume dist-rep-client-9
>121:     type protocol/client
>122:     option ping-timeout 42
>123:     option remote-host 10.70.37.215
>124:     option remote-subvolume /bricks/d5r2
>125:     option transport-type socket
>126:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>127:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>128:     option send-gids true
>129: end-volume
>130:  
>131: volume dist-rep-replicate-4
>132:     type cluster/replicate
>133:     subvolumes dist-rep-client-8 dist-rep-client-9
>134: end-volume
>135:  
>136: volume dist-rep-client-10
>137:     type protocol/client
>138:     option ping-timeout 42
>139:     option remote-host 10.70.37.44
>140:     option remote-subvolume /bricks/d6r1
>141:     option transport-type socket
>142:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>143:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>144:     option send-gids true
>145: end-volume
>146:  
>147: volume dist-rep-client-11
>148:     type protocol/client
>149:     option ping-timeout 42
>150:     option remote-host 10.70.37.201
>151:     option remote-subvolume /bricks/d6r2
>152:     option transport-type socket
>153:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>154:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>155:     option send-gids true
>156: end-volume
>157:  
>158: volume dist-rep-replicate-5
>159:     type cluster/replicate
>160:     subvolumes dist-rep-client-10 dist-rep-client-11
>161: end-volume
>162:  
>163: volume dist-rep-dht
>164:     type cluster/distribute
>165:     subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5
>166: end-volume
>167:  
>168: volume dist-rep-write-behind
>169:     type performance/write-behind
>170:     subvolumes dist-rep-dht
>171: end-volume
>172:  
>173: volume dist-rep
>174:     type debug/io-stats
>175:     option latency-measurement off
>176:     option count-fop-hits off
>177:     subvolumes dist-rep-write-behind
>178: end-volume
>179:  
>180: volume nfs-server
>181:     type nfs/server
>182:     option rpc-auth.auth-glusterfs on
>183:     option rpc-auth.auth-unix on
>184:     option rpc-auth.auth-null on
>185:     option rpc-auth.ports.insecure on
>186:     option rpc-auth-allow-insecure on
>187:     option transport-type socket
>188:     option transport.socket.listen-port 2049
>189:     option nfs.dynamic-volumes on
>190:     option nfs.nlm on
>191:     option nfs.drc on
>192:     option rpc-auth.addr.dist-rep.allow *
>193:     option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5
>194:     subvolumes dist-rep
>195: end-volume
>196:  
>+------------------------------------------------------------------------------+
>[2014-05-26 18:59:06.437988] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0)
>[2014-05-26 18:59:06.441266] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0)
>[2014-05-26 18:59:06.444752] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0)
>[2014-05-26 18:59:06.444828] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0)
>[2014-05-26 18:59:06.444864] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0)
>[2014-05-26 18:59:06.454248] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0)
>[2014-05-26 18:59:06.457394] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.457503] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0)
>[2014-05-26 18:59:06.457547] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0)
>[2014-05-26 18:59:06.457588] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0)
>[2014-05-26 18:59:06.466876] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.466988] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0)
>[2014-05-26 18:59:06.467033] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0)
>[2014-05-26 18:59:06.467075] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0)
>[2014-05-26 18:59:06.476383] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.476557] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.476646] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.476754] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'.
>[2014-05-26 18:59:06.476772] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.476880] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online.
>[2014-05-26 18:59:06.477103] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'.
>[2014-05-26 18:59:06.477121] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.477198] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online.
>[2014-05-26 18:59:06.477270] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'.
>[2014-05-26 18:59:06.477304] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.477378] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online.
>[2014-05-26 18:59:06.477418] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1
>[2014-05-26 18:59:06.477471] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.477557] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'.
>[2014-05-26 18:59:06.477572] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.477949] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1
>[2014-05-26 18:59:06.477973] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1
>[2014-05-26 18:59:06.477996] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1
>[2014-05-26 18:59:06.478014] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478072] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478137] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'.
>[2014-05-26 18:59:06.478150] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.478380] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478472] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478532] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478618] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1
>[2014-05-26 18:59:06.478645] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'.
>[2014-05-26 18:59:06.478655] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.478703] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online.
>[2014-05-26 18:59:06.478738] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 18:59:06.478801] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'.
>[2014-05-26 18:59:06.478814] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.478859] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online.
>[2014-05-26 18:59:06.479087] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'.
>[2014-05-26 18:59:06.479155] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.479322] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'.
>[2014-05-26 18:59:06.479364] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.479416] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-7' came back up; going online.
>[2014-05-26 18:59:06.479450] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1
>[2014-05-26 18:59:06.479468] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1
>[2014-05-26 18:59:06.479594] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1
>[2014-05-26 18:59:06.479893] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1
>[2014-05-26 18:59:06.479937] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'.
>[2014-05-26 18:59:06.479948] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.480353] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1
>[2014-05-26 18:59:06.481486] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'.
>[2014-05-26 18:59:06.481508] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.481808] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1
>[2014-05-26 18:59:06.484523] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'.
>[2014-05-26 18:59:06.484558] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 18:59:06.485382] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1
>[2014-05-26 18:59:06.488780] I [dht-layout.c:653:dht_layout_normalize] 0-dist-rep-dht: found anomalies in /. holes=1 overlaps=0
>[2014-05-26 19:35:35.746937] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:35:35.818682] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:35:35.822880] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:35:35.823558] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:35:57.773509] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:35:57.843737] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:35:57.847215] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:35:57.848193] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:36:47.760197] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:36:47.824062] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:36:47.827117] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:36:47.827908] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:37:43.617733] W [socket.c:522:__socket_rwv] 0-dist-rep-client-0: readv on 10.70.37.62:49153 failed (No data available)
>[2014-05-26 19:37:43.617870] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:37:43.619423] W [socket.c:522:__socket_rwv] 0-dist-rep-client-4: readv on 10.70.37.62:49154 failed (No data available)
>[2014-05-26 19:37:43.619538] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:37:43.619580] W [socket.c:522:__socket_rwv] 0-dist-rep-client-8: readv on 10.70.37.62:49155 failed (No data available)
>[2014-05-26 19:37:43.619639] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:37:58.489735] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-26 19:37:58.509596] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
>[2014-05-26 19:37:58.533978] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON
>[2014-05-26 19:37:58.539485] I [nfs.c:1353:init] 0-nfs: NFS service started
>[2014-05-26 19:37:58.539737] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-26 19:37:58.550235] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2014-05-26 19:37:58.550329] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2014-05-26 19:37:58.550381] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2014-05-26 19:37:58.550574] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.554767] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.558078] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.561376] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.564651] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.568012] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.571086] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.574171] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.577385] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.580527] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.583578] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport
>[2014-05-26 19:37:58.587275] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dist-rep-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host 10.70.37.62
>  5:     option remote-subvolume /bricks/d1r1
>  6:     option transport-type socket
>  7:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>  8:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume dist-rep-client-1
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host 10.70.37.215
> 16:     option remote-subvolume /bricks/d1r2
> 17:     option transport-type socket
> 18:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 19:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume dist-rep-replicate-0
> 24:     type cluster/replicate
> 25:     subvolumes dist-rep-client-0 dist-rep-client-1
> 26: end-volume
> 27:  
> 28: volume dist-rep-client-2
> 29:     type protocol/client
> 30:     option ping-timeout 42
> 31:     option remote-host 10.70.37.44
> 32:     option remote-subvolume /bricks/d2r1
> 33:     option transport-type socket
> 34:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 35:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 36:     option send-gids true
> 37: end-volume
> 38:  
> 39: volume dist-rep-client-3
> 40:     type protocol/client
> 41:     option ping-timeout 42
> 42:     option remote-host 10.70.37.201
> 43:     option remote-subvolume /bricks/d2r2
> 44:     option transport-type socket
> 45:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 46:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 47:     option send-gids true
> 48: end-volume
> 49:  
> 50: volume dist-rep-replicate-1
> 51:     type cluster/replicate
> 52:     subvolumes dist-rep-client-2 dist-rep-client-3
> 53: end-volume
> 54:  
> 55: volume dist-rep-client-4
> 56:     type protocol/client
> 57:     option ping-timeout 42
> 58:     option remote-host 10.70.37.62
> 59:     option remote-subvolume /bricks/d3r1
> 60:     option transport-type socket
> 61:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 62:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 63:     option send-gids true
> 64: end-volume
> 65:  
> 66: volume dist-rep-client-5
> 67:     type protocol/client
> 68:     option ping-timeout 42
> 69:     option remote-host 10.70.37.215
> 70:     option remote-subvolume /bricks/d3r2
> 71:     option transport-type socket
> 72:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 73:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 74:     option send-gids true
> 75: end-volume
> 76:  
> 77: volume dist-rep-replicate-2
> 78:     type cluster/replicate
> 79:     subvolumes dist-rep-client-4 dist-rep-client-5
> 80: end-volume
> 81:  
> 82: volume dist-rep-client-6
> 83:     type protocol/client
> 84:     option ping-timeout 42
> 85:     option remote-host 10.70.37.44
> 86:     option remote-subvolume /bricks/d4r1
> 87:     option transport-type socket
> 88:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 89:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 90:     option send-gids true
> 91: end-volume
> 92:  
> 93: volume dist-rep-client-7
> 94:     type protocol/client
> 95:     option ping-timeout 42
> 96:     option remote-host 10.70.37.201
> 97:     option remote-subvolume /bricks/d4r2
> 98:     option transport-type socket
> 99:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>100:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>101:     option send-gids true
>102: end-volume
>103:  
>104: volume dist-rep-replicate-3
>105:     type cluster/replicate
>106:     subvolumes dist-rep-client-6 dist-rep-client-7
>107: end-volume
>108:  
>109: volume dist-rep-client-8
>110:     type protocol/client
>111:     option ping-timeout 42
>112:     option remote-host 10.70.37.62
>113:     option remote-subvolume /bricks/d5r1
>114:     option transport-type socket
>115:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>116:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>117:     option send-gids true
>118: end-volume
>119:  
>120: volume dist-rep-client-9
>121:     type protocol/client
>122:     option ping-timeout 42
>123:     option remote-host 10.70.37.215
>124:     option remote-subvolume /bricks/d5r2
>125:     option transport-type socket
>126:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>127:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>128:     option send-gids true
>129: end-volume
>130:  
>131: volume dist-rep-replicate-4
>132:     type cluster/replicate
>133:     subvolumes dist-rep-client-8 dist-rep-client-9
>134: end-volume
>135:  
>136: volume dist-rep-client-10
>137:     type protocol/client
>138:     option ping-timeout 42
>139:     option remote-host 10.70.37.44
>140:     option remote-subvolume /bricks/d6r1
>141:     option transport-type socket
>142:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>143:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>144:     option send-gids true
>145: end-volume
>146:  
>147: volume dist-rep-client-11
>148:     type protocol/client
>149:     option ping-timeout 42
>150:     option remote-host 10.70.37.201
>151:     option remote-subvolume /bricks/d6r2
>152:     option transport-type socket
>153:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>154:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>155:     option send-gids true
>156: end-volume
>157:  
>158: volume dist-rep-replicate-5
>159:     type cluster/replicate
>160:     subvolumes dist-rep-client-10 dist-rep-client-11
>161: end-volume
>162:  
>163: volume dist-rep-dht
>164:     type cluster/distribute
>165:     subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5
>166: end-volume
>167:  
>168: volume dist-rep-write-behind
>169:     type performance/write-behind
>170:     subvolumes dist-rep-dht
>171: end-volume
>172:  
>173: volume dist-rep
>174:     type debug/io-stats
>175:     option latency-measurement off
>176:     option count-fop-hits off
>177:     subvolumes dist-rep-write-behind
>178: end-volume
>179:  
>180: volume nfs-server
>181:     type nfs/server
>182:     option rpc-auth.auth-glusterfs on
>183:     option rpc-auth.auth-unix on
>184:     option rpc-auth.auth-null on
>185:     option rpc-auth.ports.insecure on
>186:     option rpc-auth-allow-insecure on
>187:     option transport-type socket
>188:     option transport.socket.listen-port 2049
>189:     option nfs.dynamic-volumes on
>190:     option nfs.nlm on
>191:     option nfs.drc on
>192:     option rpc-auth.addr.dist-rep.allow *
>193:     option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5
>194:     subvolumes dist-rep
>195: end-volume
>196:  
>+------------------------------------------------------------------------------+
>[2014-05-26 19:37:58.592844] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0)
>[2014-05-26 19:37:58.593007] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0)
>[2014-05-26 19:37:58.593079] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0)
>[2014-05-26 19:37:58.606127] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.606240] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.606319] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.606720] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'.
>[2014-05-26 19:37:58.606743] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.606828] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online.
>[2014-05-26 19:37:58.606870] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'.
>[2014-05-26 19:37:58.606881] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.606970] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online.
>[2014-05-26 19:37:58.607000] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1
>[2014-05-26 19:37:58.607022] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'.
>[2014-05-26 19:37:58.607031] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.607122] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online.
>[2014-05-26 19:37:58.607167] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1
>[2014-05-26 19:37:58.607350] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1
>[2014-05-26 19:37:58.630217] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0)
>[2014-05-26 19:37:58.636469] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0)
>[2014-05-26 19:37:58.636539] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0)
>[2014-05-26 19:37:58.648113] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.648635] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'.
>[2014-05-26 19:37:58.648660] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.648761] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online.
>[2014-05-26 19:37:58.649111] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0)
>[2014-05-26 19:37:58.649173] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0)
>[2014-05-26 19:37:58.649229] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0)
>[2014-05-26 19:37:58.649287] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1
>[2014-05-26 19:37:58.664693] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.664804] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0)
>[2014-05-26 19:37:58.664904] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0)
>[2014-05-26 19:37:58.664985] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0)
>[2014-05-26 19:37:58.665048] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.665289] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'.
>[2014-05-26 19:37:58.665334] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.665483] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-6' came back up; going online.
>[2014-05-26 19:37:58.678107] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1
>[2014-05-26 19:37:58.678143] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.678231] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.678333] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'.
>[2014-05-26 19:37:58.678349] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.678399] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online.
>[2014-05-26 19:37:58.678588] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'.
>[2014-05-26 19:37:58.678605] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.679070] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1
>[2014-05-26 19:37:58.679098] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.679167] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.679298] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'.
>[2014-05-26 19:37:58.679313] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.679490] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'.
>[2014-05-26 19:37:58.679520] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.679690] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.679771] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1
>[2014-05-26 19:37:58.679838] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'.
>[2014-05-26 19:37:58.679852] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.680037] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1
>[2014-05-26 19:37:58.680088] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1
>[2014-05-26 19:37:58.680108] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:37:58.680476] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1
>[2014-05-26 19:37:58.680796] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'.
>[2014-05-26 19:37:58.680811] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.681098] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1
>[2014-05-26 19:37:58.681913] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'.
>[2014-05-26 19:37:58.681930] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:37:58.682402] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1
>[2014-05-26 19:42:04.841386] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available)
>[2014-05-26 19:42:12.527990] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-26 19:42:13.542756] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-26 19:42:14.566862] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
>[2014-05-26 19:42:14.628849] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON
>[2014-05-26 19:42:14.634054] I [nfs.c:1353:init] 0-nfs: NFS service started
>[2014-05-26 19:42:14.634405] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-26 19:42:14.645448] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2014-05-26 19:42:14.645547] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2014-05-26 19:42:14.645598] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2014-05-26 19:42:14.645771] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.649428] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.652819] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.655856] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.658814] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.661786] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.664760] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.667702] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.670662] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.673640] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.676613] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport
>[2014-05-26 19:42:14.679558] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dist-rep-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host 10.70.37.62
>  5:     option remote-subvolume /bricks/d1r1
>  6:     option transport-type socket
>  7:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>  8:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume dist-rep-client-1
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host 10.70.37.215
> 16:     option remote-subvolume /bricks/d1r2
> 17:     option transport-type socket
> 18:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 19:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume dist-rep-replicate-0
> 24:     type cluster/replicate
> 25:     subvolumes dist-rep-client-0 dist-rep-client-1
> 26: end-volume
> 27:  
> 28: volume dist-rep-client-2
> 29:     type protocol/client
> 30:     option ping-timeout 42
> 31:     option remote-host 10.70.37.44
> 32:     option remote-subvolume /bricks/d2r1
> 33:     option transport-type socket
> 34:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 35:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 36:     option send-gids true
> 37: end-volume
> 38:  
> 39: volume dist-rep-client-3
> 40:     type protocol/client
> 41:     option ping-timeout 42
> 42:     option remote-host 10.70.37.201
> 43:     option remote-subvolume /bricks/d2r2
> 44:     option transport-type socket
> 45:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 46:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 47:     option send-gids true
> 48: end-volume
> 49:  
> 50: volume dist-rep-replicate-1
> 51:     type cluster/replicate
> 52:     subvolumes dist-rep-client-2 dist-rep-client-3
> 53: end-volume
> 54:  
> 55: volume dist-rep-client-4
> 56:     type protocol/client
> 57:     option ping-timeout 42
> 58:     option remote-host 10.70.37.62
> 59:     option remote-subvolume /bricks/d3r1
> 60:     option transport-type socket
> 61:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 62:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 63:     option send-gids true
> 64: end-volume
> 65:  
> 66: volume dist-rep-client-5
> 67:     type protocol/client
> 68:     option ping-timeout 42
> 69:     option remote-host 10.70.37.215
> 70:     option remote-subvolume /bricks/d3r2
> 71:     option transport-type socket
> 72:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 73:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 74:     option send-gids true
> 75: end-volume
> 76:  
> 77: volume dist-rep-replicate-2
> 78:     type cluster/replicate
> 79:     subvolumes dist-rep-client-4 dist-rep-client-5
> 80: end-volume
> 81:  
> 82: volume dist-rep-client-6
> 83:     type protocol/client
> 84:     option ping-timeout 42
> 85:     option remote-host 10.70.37.44
> 86:     option remote-subvolume /bricks/d4r1
> 87:     option transport-type socket
> 88:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 89:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 90:     option send-gids true
> 91: end-volume
> 92:  
> 93: volume dist-rep-client-7
> 94:     type protocol/client
> 95:     option ping-timeout 42
> 96:     option remote-host 10.70.37.201
> 97:     option remote-subvolume /bricks/d4r2
> 98:     option transport-type socket
> 99:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>100:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>101:     option send-gids true
>102: end-volume
>103:  
>104: volume dist-rep-replicate-3
>105:     type cluster/replicate
>106:     subvolumes dist-rep-client-6 dist-rep-client-7
>107: end-volume
>108:  
>109: volume dist-rep-client-8
>110:     type protocol/client
>111:     option ping-timeout 42
>112:     option remote-host 10.70.37.62
>113:     option remote-subvolume /bricks/d5r1
>114:     option transport-type socket
>115:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>116:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>117:     option send-gids true
>118: end-volume
>119:  
>120: volume dist-rep-client-9
>121:     type protocol/client
>122:     option ping-timeout 42
>123:     option remote-host 10.70.37.215
>124:     option remote-subvolume /bricks/d5r2
>125:     option transport-type socket
>126:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>127:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>128:     option send-gids true
>129: end-volume
>130:  
>131: volume dist-rep-replicate-4
>132:     type cluster/replicate
>133:     subvolumes dist-rep-client-8 dist-rep-client-9
>134: end-volume
>135:  
>136: volume dist-rep-client-10
>137:     type protocol/client
>138:     option ping-timeout 42
>139:     option remote-host 10.70.37.44
>140:     option remote-subvolume /bricks/d6r1
>141:     option transport-type socket
>142:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>143:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>144:     option send-gids true
>145: end-volume
>146:  
>147: volume dist-rep-client-11
>148:     type protocol/client
>149:     option ping-timeout 42
>150:     option remote-host 10.70.37.201
>151:     option remote-subvolume /bricks/d6r2
>152:     option transport-type socket
>153:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>154:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>155:     option send-gids true
>156: end-volume
>157:  
>158: volume dist-rep-replicate-5
>159:     type cluster/replicate
>160:     subvolumes dist-rep-client-10 dist-rep-client-11
>161: end-volume
>162:  
>163: volume dist-rep-dht
>164:     type cluster/distribute
>165:     subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5
>166: end-volume
>167:  
>168: volume dist-rep-write-behind
>169:     type performance/write-behind
>170:     subvolumes dist-rep-dht
>171: end-volume
>172:  
>173: volume dist-rep
>174:     type debug/io-stats
>175:     option latency-measurement off
>176:     option count-fop-hits off
>177:     subvolumes dist-rep-write-behind
>178: end-volume
>179:  
>180: volume nfs-server
>181:     type nfs/server
>182:     option rpc-auth.auth-glusterfs on
>183:     option rpc-auth.auth-unix on
>184:     option rpc-auth.auth-null on
>185:     option rpc-auth.ports.insecure on
>186:     option rpc-auth-allow-insecure on
>187:     option transport-type socket
>188:     option transport.socket.listen-port 2049
>189:     option nfs.dynamic-volumes on
>190:     option nfs.nlm on
>191:     option nfs.drc on
>192:     option rpc-auth.addr.dist-rep.allow *
>193:     option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5
>194:     subvolumes dist-rep
>195: end-volume
>196:  
>+------------------------------------------------------------------------------+
>[2014-05-26 19:42:14.684290] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:14.684372] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:14.684399] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:14.684445] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:14.684477] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:14.684494] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:14.684890] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0)
>[2014-05-26 19:42:14.688058] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0)
>[2014-05-26 19:42:14.688137] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0)
>[2014-05-26 19:42:14.688233] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0)
>[2014-05-26 19:42:14.697283] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0)
>[2014-05-26 19:42:14.697331] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0)
>[2014-05-26 19:42:14.703493] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.703564] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.703621] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.704141] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'.
>[2014-05-26 19:42:14.704182] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.704238] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online.
>[2014-05-26 19:42:14.704295] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'.
>[2014-05-26 19:42:14.704308] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.704358] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online.
>[2014-05-26 19:42:14.704608] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.704675] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1
>[2014-05-26 19:42:14.704863] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1
>[2014-05-26 19:42:14.704979] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.705175] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'.
>[2014-05-26 19:42:14.705191] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.705247] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-6' came back up; going online.
>[2014-05-26 19:42:14.705276] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:14.705587] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1
>[2014-05-26 19:42:14.705757] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'.
>[2014-05-26 19:42:14.705772] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.705927] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'.
>[2014-05-26 19:42:14.705942] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.706311] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1
>[2014-05-26 19:42:14.706572] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1
>[2014-05-26 19:42:14.706820] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'.
>[2014-05-26 19:42:14.706842] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:14.707334] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1
>[2014-05-26 19:42:15.749343] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:15.749445] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-1: disconnected from dist-rep-client-1. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:15.749466] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-05-26 19:42:15.749580] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:15.749676] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-05-26 19:42:15.749780] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-5: disconnected from dist-rep-client-5. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:15.749805] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-05-26 19:42:15.749831] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-9: disconnected from dist-rep-client-9. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 19:42:15.749856] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-4: All subvolumes are down. Going offline until atleast one of them comes back up.
>[2014-05-26 19:42:15.756188] I [dht-layout.c:653:dht_layout_normalize] 0-dist-rep-dht: found anomalies in /. holes=3 overlaps=0
>[2014-05-26 19:42:15.756271] W [dht-selfheal.c:952:dht_selfheal_directory] 0-dist-rep-dht: 3 subvolumes down -- not fixing
>[2014-05-26 19:42:18.552152] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0)
>[2014-05-26 19:42:18.561550] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:18.562012] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'.
>[2014-05-26 19:42:18.562044] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:18.562327] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online.
>[2014-05-26 19:42:18.562565] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1
>[2014-05-26 19:42:18.565790] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0)
>[2014-05-26 19:42:18.578994] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:18.579311] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'.
>[2014-05-26 19:42:18.579331] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:18.579403] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online.
>[2014-05-26 19:42:18.579499] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1
>[2014-05-26 19:42:18.579572] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0)
>[2014-05-26 19:42:18.586281] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:18.586727] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'.
>[2014-05-26 19:42:18.586750] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:18.586829] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online.
>[2014-05-26 19:42:18.587026] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1
>[2014-05-26 19:42:21.876433] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0)
>[2014-05-26 19:42:21.882538] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:21.883038] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'.
>[2014-05-26 19:42:21.883086] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:21.883338] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1
>[2014-05-26 19:42:24.593551] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0)
>[2014-05-26 19:42:24.601229] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:24.601321] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0)
>[2014-05-26 19:42:24.610727] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'.
>[2014-05-26 19:42:24.610751] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:24.611067] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1
>[2014-05-26 19:42:24.611126] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 19:42:24.611536] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'.
>[2014-05-26 19:42:24.611560] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 19:42:24.611812] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1
>[2014-05-26 19:46:26.001795] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:46:26.064259] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:46:26.064800] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:46:26.068462] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:56:26.793882] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:56:26.862455] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 19:56:26.865036] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 19:56:26.865135] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:12:34.579086] W [socket.c:522:__socket_rwv] 0-dist-rep-client-4: readv on 10.70.37.62:49154 failed (No data available)
>[2014-05-26 20:12:34.579193] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 20:12:34.579506] W [socket.c:522:__socket_rwv] 0-dist-rep-client-0: readv on 10.70.37.62:49153 failed (No data available)
>[2014-05-26 20:12:34.579625] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 20:12:34.579671] W [socket.c:522:__socket_rwv] 0-dist-rep-client-8: readv on 10.70.37.62:49155 failed (No data available)
>[2014-05-26 20:12:34.579736] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-05-26 20:23:53.454547] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket)
>[2014-05-26 20:23:53.474993] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
>[2014-05-26 20:23:53.500618] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON
>[2014-05-26 20:23:53.506231] I [nfs.c:1353:init] 0-nfs: NFS service started
>[2014-05-26 20:23:53.506470] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>[2014-05-26 20:23:53.516037] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>[2014-05-26 20:23:53.516129] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>[2014-05-26 20:23:53.516185] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized
>[2014-05-26 20:23:53.516357] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.519520] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.522553] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.525575] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.528613] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.531904] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.534942] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.537935] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.540906] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.544041] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.547111] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport
>[2014-05-26 20:23:53.550079] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume dist-rep-client-0
>  2:     type protocol/client
>  3:     option ping-timeout 42
>  4:     option remote-host 10.70.37.62
>  5:     option remote-subvolume /bricks/d1r1
>  6:     option transport-type socket
>  7:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>  8:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>  9:     option send-gids true
> 10: end-volume
> 11:  
> 12: volume dist-rep-client-1
> 13:     type protocol/client
> 14:     option ping-timeout 42
> 15:     option remote-host 10.70.37.215
> 16:     option remote-subvolume /bricks/d1r2
> 17:     option transport-type socket
> 18:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 19:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 20:     option send-gids true
> 21: end-volume
> 22:  
> 23: volume dist-rep-replicate-0
> 24:     type cluster/replicate
> 25:     subvolumes dist-rep-client-0 dist-rep-client-1
> 26: end-volume
> 27:  
> 28: volume dist-rep-client-2
> 29:     type protocol/client
> 30:     option ping-timeout 42
> 31:     option remote-host 10.70.37.44
> 32:     option remote-subvolume /bricks/d2r1
> 33:     option transport-type socket
> 34:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 35:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 36:     option send-gids true
> 37: end-volume
> 38:  
> 39: volume dist-rep-client-3
> 40:     type protocol/client
> 41:     option ping-timeout 42
> 42:     option remote-host 10.70.37.201
> 43:     option remote-subvolume /bricks/d2r2
> 44:     option transport-type socket
> 45:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 46:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 47:     option send-gids true
> 48: end-volume
> 49:  
> 50: volume dist-rep-replicate-1
> 51:     type cluster/replicate
> 52:     subvolumes dist-rep-client-2 dist-rep-client-3
> 53: end-volume
> 54:  
> 55: volume dist-rep-client-4
> 56:     type protocol/client
> 57:     option ping-timeout 42
> 58:     option remote-host 10.70.37.62
> 59:     option remote-subvolume /bricks/d3r1
> 60:     option transport-type socket
> 61:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 62:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 63:     option send-gids true
> 64: end-volume
> 65:  
> 66: volume dist-rep-client-5
> 67:     type protocol/client
> 68:     option ping-timeout 42
> 69:     option remote-host 10.70.37.215
> 70:     option remote-subvolume /bricks/d3r2
> 71:     option transport-type socket
> 72:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 73:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 74:     option send-gids true
> 75: end-volume
> 76:  
> 77: volume dist-rep-replicate-2
> 78:     type cluster/replicate
> 79:     subvolumes dist-rep-client-4 dist-rep-client-5
> 80: end-volume
> 81:  
> 82: volume dist-rep-client-6
> 83:     type protocol/client
> 84:     option ping-timeout 42
> 85:     option remote-host 10.70.37.44
> 86:     option remote-subvolume /bricks/d4r1
> 87:     option transport-type socket
> 88:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
> 89:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
> 90:     option send-gids true
> 91: end-volume
> 92:  
> 93: volume dist-rep-client-7
> 94:     type protocol/client
> 95:     option ping-timeout 42
> 96:     option remote-host 10.70.37.201
> 97:     option remote-subvolume /bricks/d4r2
> 98:     option transport-type socket
> 99:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>100:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>101:     option send-gids true
>102: end-volume
>103:  
>104: volume dist-rep-replicate-3
>105:     type cluster/replicate
>106:     subvolumes dist-rep-client-6 dist-rep-client-7
>107: end-volume
>108:  
>109: volume dist-rep-client-8
>110:     type protocol/client
>111:     option ping-timeout 42
>112:     option remote-host 10.70.37.62
>113:     option remote-subvolume /bricks/d5r1
>114:     option transport-type socket
>115:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>116:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>117:     option send-gids true
>118: end-volume
>119:  
>120: volume dist-rep-client-9
>121:     type protocol/client
>122:     option ping-timeout 42
>123:     option remote-host 10.70.37.215
>124:     option remote-subvolume /bricks/d5r2
>125:     option transport-type socket
>126:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>127:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>128:     option send-gids true
>129: end-volume
>130:  
>131: volume dist-rep-replicate-4
>132:     type cluster/replicate
>133:     subvolumes dist-rep-client-8 dist-rep-client-9
>134: end-volume
>135:  
>136: volume dist-rep-client-10
>137:     type protocol/client
>138:     option ping-timeout 42
>139:     option remote-host 10.70.37.44
>140:     option remote-subvolume /bricks/d6r1
>141:     option transport-type socket
>142:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>143:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>144:     option send-gids true
>145: end-volume
>146:  
>147: volume dist-rep-client-11
>148:     type protocol/client
>149:     option ping-timeout 42
>150:     option remote-host 10.70.37.201
>151:     option remote-subvolume /bricks/d6r2
>152:     option transport-type socket
>153:     option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9
>154:     option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770
>155:     option send-gids true
>156: end-volume
>157:  
>158: volume dist-rep-replicate-5
>159:     type cluster/replicate
>160:     subvolumes dist-rep-client-10 dist-rep-client-11
>161: end-volume
>162:  
>163: volume dist-rep-dht
>164:     type cluster/distribute
>165:     subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5
>166: end-volume
>167:  
>168: volume dist-rep-write-behind
>169:     type performance/write-behind
>170:     subvolumes dist-rep-dht
>171: end-volume
>172:  
>173: volume dist-rep
>174:     type debug/io-stats
>175:     option latency-measurement off
>176:     option count-fop-hits off
>177:     subvolumes dist-rep-write-behind
>178: end-volume
>179:  
>180: volume nfs-server
>181:     type nfs/server
>182:     option rpc-auth.auth-glusterfs on
>183:     option rpc-auth.auth-unix on
>184:     option rpc-auth.auth-null on
>185:     option rpc-auth.ports.insecure on
>186:     option rpc-auth-allow-insecure on
>187:     option transport-type socket
>188:     option transport.socket.listen-port 2049
>189:     option nfs.dynamic-volumes on
>190:     option nfs.nlm on
>191:     option nfs.drc on
>192:     option rpc-auth.addr.dist-rep.allow *
>193:     option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5
>194:     subvolumes dist-rep
>195: end-volume
>196:  
>+------------------------------------------------------------------------------+
>[2014-05-26 20:23:53.555077] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0)
>[2014-05-26 20:23:53.558365] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0)
>[2014-05-26 20:23:53.558420] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0)
>[2014-05-26 20:23:53.565130] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.565228] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.565316] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.565406] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'.
>[2014-05-26 20:23:53.565428] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.565518] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online.
>[2014-05-26 20:23:53.565612] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1
>[2014-05-26 20:23:53.568313] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'.
>[2014-05-26 20:23:53.568331] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.568376] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online.
>[2014-05-26 20:23:53.568497] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1
>[2014-05-26 20:23:53.568655] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'.
>[2014-05-26 20:23:53.568671] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.568712] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online.
>[2014-05-26 20:23:53.568859] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1
>[2014-05-26 20:23:53.594410] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0)
>[2014-05-26 20:23:53.600593] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0)
>[2014-05-26 20:23:53.600820] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0)
>[2014-05-26 20:23:53.600920] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0)
>[2014-05-26 20:23:53.601025] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0)
>[2014-05-26 20:23:53.615483] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0)
>[2014-05-26 20:23:53.615577] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0)
>[2014-05-26 20:23:53.615672] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0)
>[2014-05-26 20:23:53.615722] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0)
>[2014-05-26 20:23:53.628243] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.628342] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.628478] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.628570] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.628798] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.628884] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'.
>[2014-05-26 20:23:53.628899] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.629173] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'.
>[2014-05-26 20:23:53.629191] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.629251] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-3' came back up; going online.
>[2014-05-26 20:23:53.629353] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'.
>[2014-05-26 20:23:53.629368] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.629709] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.629796] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'.
>[2014-05-26 20:23:53.629819] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.629894] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-7' came back up; going online.
>[2014-05-26 20:23:53.629934] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.630015] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1
>[2014-05-26 20:23:53.630060] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.630120] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1
>[2014-05-26 20:23:53.630163] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-05-26 20:23:53.630215] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1
>[2014-05-26 20:23:53.630327] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1
>[2014-05-26 20:23:53.630651] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'.
>[2014-05-26 20:23:53.630667] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.631177] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'.
>[2014-05-26 20:23:53.631200] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.631354] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1
>[2014-05-26 20:23:53.632189] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1
>[2014-05-26 20:23:53.632247] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'.
>[2014-05-26 20:23:53.632261] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.632546] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1
>[2014-05-26 20:23:53.633622] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'.
>[2014-05-26 20:23:53.633642] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.633690] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online.
>[2014-05-26 20:23:53.634043] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1
>[2014-05-26 20:23:53.634244] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'.
>[2014-05-26 20:23:53.634259] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2014-05-26 20:23:53.634698] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1
>[2014-05-26 20:45:30.499777] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:45:30.500682] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:47:36.560612] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:47:36.561694] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:47:48.296437] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:47:48.298327] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:48:26.647506] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:48:26.712162] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:48:26.716245] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:48:26.716990] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:48:46.056673] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:48:46.121892] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:48:46.126373] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:48:46.126433] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:49:25.908758] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:49:25.909521] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:58:47.058336] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:58:47.059406] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:58:53.439086] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:58:53.509725] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:58:53.514085] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:58:53.514178] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:58:56.658559] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:58:56.720248] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:58:56.726084] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:58:56.726122] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:59:03.738805] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:59:03.739852] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:59:08.975778] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:59:08.976858] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null)
>[2014-05-26 20:59:24.798614] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:59:24.860331] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:59:24.863680] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:59:24.864117] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:59:46.004545] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:59:46.070053] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-05-26 20:59:46.074438] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-05-26 20:59:46.074666] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing

Actions: View

Attachments on bug 1101438: 899401 | 954119 | 954361