Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 899401 Details for
Bug 1101438
nfs: ls shows "Permission denied" with root-squash
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
nfs logs
nfs.log (text/x-log), 98.42 KB, created by
Saurabh
on 2014-05-27 08:03:09 UTC
(
hide
)
Description:
nfs logs
Filename:
MIME Type:
Creator:
Saurabh
Created:
2014-05-27 08:03:09 UTC
Size:
98.42 KB
patch
obsolete
>[2014-05-25 22:40:06.405041] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-25 22:40:06.422615] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-05-25 22:40:06.446251] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON >[2014-05-25 22:40:06.451251] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-05-25 22:40:06.451586] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-25 22:40:06.452507] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-05-25 22:40:06.452575] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-05-25 22:40:06.452625] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-05-25 22:40:06.452698] I [client.c:2280:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume testvol-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.37.62 > 5: option remote-subvolume /tmp/brick > 6: option transport-type socket > 7: option username 6c4a83e9-23e7-47f7-ae92-dec72d4ee6a8 > 8: option password 4a706d27-89b6-4b6d-87d4-48cc949815c9 > 9: option send-gids true > 10: end-volume > 11: > 12: volume testvol-dht > 13: type cluster/distribute > 14: subvolumes testvol-client-0 > 15: end-volume > 16: > 17: volume testvol-write-behind > 18: type performance/write-behind > 19: subvolumes testvol-dht > 20: end-volume > 21: > 22: volume testvol > 23: type debug/io-stats > 24: option latency-measurement off > 25: option count-fop-hits off > 26: subvolumes testvol-write-behind > 27: end-volume > 28: > 29: volume nfs-server > 30: type nfs/server > 31: option rpc-auth.auth-glusterfs on > 32: option rpc-auth.auth-unix on > 33: option rpc-auth.auth-null on > 34: option rpc-auth.ports.insecure on > 35: option rpc-auth-allow-insecure on > 36: option transport-type socket > 37: option transport.socket.listen-port 2049 > 38: option nfs.dynamic-volumes on > 39: option nfs.nlm on > 40: option nfs.drc on > 41: option rpc-auth.addr.testvol.allow * > 42: option nfs3.testvol.volume-id 036848d4-e54d-4606-b21f-bb0e1d380bc0 > 43: subvolumes testvol > 44: end-volume > 45: >+------------------------------------------------------------------------------+ >[2014-05-25 22:40:06.457016] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49152 (from 0) >[2014-05-25 22:40:06.460396] I [client-handshake.c:1415:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-25 22:40:06.463470] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/tmp/brick'. >[2014-05-25 22:40:06.463514] I [client-handshake.c:1212:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-25 22:40:06.463822] I [client-handshake.c:188:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1 >[2014-05-25 22:40:06.464591] I [dht-layout.c:653:dht_layout_normalize] 0-testvol-dht: found anomalies in /. holes=1 overlaps=0 >[2014-05-25 22:40:39.697322] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:40:39.730476] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:40:39.735279] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-25 22:40:39.735682] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:51:22.735395] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:51:22.759200] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:51:49.193918] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:51:49.220479] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:52:19.510215] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:52:19.533004] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:54:17.474515] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:54:17.511561] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:54:42.459596] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:54:42.484899] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 22:54:51.958674] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:54:51.990855] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 22:54:51.995246] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-25 22:54:51.995546] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 23:03:36.635944] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 23:03:36.668751] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 23:03:43.692258] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 23:03:43.715562] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 23:40:12.355766] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 23:40:12.381863] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-25 23:41:46.429087] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-25 23:41:46.452264] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 18:54:12.571391] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 18:54:12.606083] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 18:54:12.609921] I [dht-shared.c:332:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-26 18:54:12.610198] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 18:54:36.622463] W [socket.c:522:__socket_rwv] 0-testvol-client-0: readv on 10.70.37.62:49152 failed (No data available) >[2014-05-26 18:54:36.622679] I [client.c:2215:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 18:59:06.283180] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-26 18:59:06.305907] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-05-26 18:59:06.373445] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON >[2014-05-26 18:59:06.379214] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-05-26 18:59:06.379490] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-26 18:59:06.390941] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-05-26 18:59:06.391073] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-05-26 18:59:06.391144] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-05-26 18:59:06.391341] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.394994] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.398282] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.401601] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.404687] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.407810] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.415946] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.421056] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.424093] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.427183] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.430162] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport >[2014-05-26 18:59:06.433157] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-rep-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.37.62 > 5: option remote-subvolume /bricks/d1r1 > 6: option transport-type socket > 7: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 8: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-rep-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.37.215 > 16: option remote-subvolume /bricks/d1r2 > 17: option transport-type socket > 18: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 19: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-rep-replicate-0 > 24: type cluster/replicate > 25: subvolumes dist-rep-client-0 dist-rep-client-1 > 26: end-volume > 27: > 28: volume dist-rep-client-2 > 29: type protocol/client > 30: option ping-timeout 42 > 31: option remote-host 10.70.37.44 > 32: option remote-subvolume /bricks/d2r1 > 33: option transport-type socket > 34: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 35: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 36: option send-gids true > 37: end-volume > 38: > 39: volume dist-rep-client-3 > 40: type protocol/client > 41: option ping-timeout 42 > 42: option remote-host 10.70.37.201 > 43: option remote-subvolume /bricks/d2r2 > 44: option transport-type socket > 45: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 46: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 47: option send-gids true > 48: end-volume > 49: > 50: volume dist-rep-replicate-1 > 51: type cluster/replicate > 52: subvolumes dist-rep-client-2 dist-rep-client-3 > 53: end-volume > 54: > 55: volume dist-rep-client-4 > 56: type protocol/client > 57: option ping-timeout 42 > 58: option remote-host 10.70.37.62 > 59: option remote-subvolume /bricks/d3r1 > 60: option transport-type socket > 61: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 62: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 63: option send-gids true > 64: end-volume > 65: > 66: volume dist-rep-client-5 > 67: type protocol/client > 68: option ping-timeout 42 > 69: option remote-host 10.70.37.215 > 70: option remote-subvolume /bricks/d3r2 > 71: option transport-type socket > 72: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 73: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 74: option send-gids true > 75: end-volume > 76: > 77: volume dist-rep-replicate-2 > 78: type cluster/replicate > 79: subvolumes dist-rep-client-4 dist-rep-client-5 > 80: end-volume > 81: > 82: volume dist-rep-client-6 > 83: type protocol/client > 84: option ping-timeout 42 > 85: option remote-host 10.70.37.44 > 86: option remote-subvolume /bricks/d4r1 > 87: option transport-type socket > 88: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 89: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 90: option send-gids true > 91: end-volume > 92: > 93: volume dist-rep-client-7 > 94: type protocol/client > 95: option ping-timeout 42 > 96: option remote-host 10.70.37.201 > 97: option remote-subvolume /bricks/d4r2 > 98: option transport-type socket > 99: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >100: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >101: option send-gids true >102: end-volume >103: >104: volume dist-rep-replicate-3 >105: type cluster/replicate >106: subvolumes dist-rep-client-6 dist-rep-client-7 >107: end-volume >108: >109: volume dist-rep-client-8 >110: type protocol/client >111: option ping-timeout 42 >112: option remote-host 10.70.37.62 >113: option remote-subvolume /bricks/d5r1 >114: option transport-type socket >115: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >116: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >117: option send-gids true >118: end-volume >119: >120: volume dist-rep-client-9 >121: type protocol/client >122: option ping-timeout 42 >123: option remote-host 10.70.37.215 >124: option remote-subvolume /bricks/d5r2 >125: option transport-type socket >126: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >127: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >128: option send-gids true >129: end-volume >130: >131: volume dist-rep-replicate-4 >132: type cluster/replicate >133: subvolumes dist-rep-client-8 dist-rep-client-9 >134: end-volume >135: >136: volume dist-rep-client-10 >137: type protocol/client >138: option ping-timeout 42 >139: option remote-host 10.70.37.44 >140: option remote-subvolume /bricks/d6r1 >141: option transport-type socket >142: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >143: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >144: option send-gids true >145: end-volume >146: >147: volume dist-rep-client-11 >148: type protocol/client >149: option ping-timeout 42 >150: option remote-host 10.70.37.201 >151: option remote-subvolume /bricks/d6r2 >152: option transport-type socket >153: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >154: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >155: option send-gids true >156: end-volume >157: >158: volume dist-rep-replicate-5 >159: type cluster/replicate >160: subvolumes dist-rep-client-10 dist-rep-client-11 >161: end-volume >162: >163: volume dist-rep-dht >164: type cluster/distribute >165: subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5 >166: end-volume >167: >168: volume dist-rep-write-behind >169: type performance/write-behind >170: subvolumes dist-rep-dht >171: end-volume >172: >173: volume dist-rep >174: type debug/io-stats >175: option latency-measurement off >176: option count-fop-hits off >177: subvolumes dist-rep-write-behind >178: end-volume >179: >180: volume nfs-server >181: type nfs/server >182: option rpc-auth.auth-glusterfs on >183: option rpc-auth.auth-unix on >184: option rpc-auth.auth-null on >185: option rpc-auth.ports.insecure on >186: option rpc-auth-allow-insecure on >187: option transport-type socket >188: option transport.socket.listen-port 2049 >189: option nfs.dynamic-volumes on >190: option nfs.nlm on >191: option nfs.drc on >192: option rpc-auth.addr.dist-rep.allow * >193: option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5 >194: subvolumes dist-rep >195: end-volume >196: >+------------------------------------------------------------------------------+ >[2014-05-26 18:59:06.437988] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0) >[2014-05-26 18:59:06.441266] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0) >[2014-05-26 18:59:06.444752] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0) >[2014-05-26 18:59:06.444828] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0) >[2014-05-26 18:59:06.444864] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0) >[2014-05-26 18:59:06.454248] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0) >[2014-05-26 18:59:06.457394] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.457503] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0) >[2014-05-26 18:59:06.457547] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0) >[2014-05-26 18:59:06.457588] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0) >[2014-05-26 18:59:06.466876] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.466988] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0) >[2014-05-26 18:59:06.467033] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0) >[2014-05-26 18:59:06.467075] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0) >[2014-05-26 18:59:06.476383] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.476557] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.476646] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.476754] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'. >[2014-05-26 18:59:06.476772] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.476880] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online. >[2014-05-26 18:59:06.477103] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'. >[2014-05-26 18:59:06.477121] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.477198] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online. >[2014-05-26 18:59:06.477270] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'. >[2014-05-26 18:59:06.477304] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.477378] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online. >[2014-05-26 18:59:06.477418] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1 >[2014-05-26 18:59:06.477471] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.477557] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'. >[2014-05-26 18:59:06.477572] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.477949] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1 >[2014-05-26 18:59:06.477973] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1 >[2014-05-26 18:59:06.477996] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1 >[2014-05-26 18:59:06.478014] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478072] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478137] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'. >[2014-05-26 18:59:06.478150] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.478380] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478472] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478532] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478618] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1 >[2014-05-26 18:59:06.478645] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'. >[2014-05-26 18:59:06.478655] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.478703] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online. >[2014-05-26 18:59:06.478738] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 18:59:06.478801] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'. >[2014-05-26 18:59:06.478814] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.478859] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online. >[2014-05-26 18:59:06.479087] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'. >[2014-05-26 18:59:06.479155] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.479322] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'. >[2014-05-26 18:59:06.479364] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.479416] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-7' came back up; going online. >[2014-05-26 18:59:06.479450] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1 >[2014-05-26 18:59:06.479468] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1 >[2014-05-26 18:59:06.479594] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1 >[2014-05-26 18:59:06.479893] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1 >[2014-05-26 18:59:06.479937] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'. >[2014-05-26 18:59:06.479948] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.480353] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1 >[2014-05-26 18:59:06.481486] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'. >[2014-05-26 18:59:06.481508] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.481808] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1 >[2014-05-26 18:59:06.484523] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'. >[2014-05-26 18:59:06.484558] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 18:59:06.485382] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1 >[2014-05-26 18:59:06.488780] I [dht-layout.c:653:dht_layout_normalize] 0-dist-rep-dht: found anomalies in /. holes=1 overlaps=0 >[2014-05-26 19:35:35.746937] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:35:35.818682] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:35:35.822880] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:35:35.823558] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:35:57.773509] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:35:57.843737] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:35:57.847215] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:35:57.848193] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:36:47.760197] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:36:47.824062] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:36:47.827117] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:36:47.827908] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:37:43.617733] W [socket.c:522:__socket_rwv] 0-dist-rep-client-0: readv on 10.70.37.62:49153 failed (No data available) >[2014-05-26 19:37:43.617870] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:37:43.619423] W [socket.c:522:__socket_rwv] 0-dist-rep-client-4: readv on 10.70.37.62:49154 failed (No data available) >[2014-05-26 19:37:43.619538] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:37:43.619580] W [socket.c:522:__socket_rwv] 0-dist-rep-client-8: readv on 10.70.37.62:49155 failed (No data available) >[2014-05-26 19:37:43.619639] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:37:58.489735] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-26 19:37:58.509596] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-05-26 19:37:58.533978] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON >[2014-05-26 19:37:58.539485] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-05-26 19:37:58.539737] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-26 19:37:58.550235] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-05-26 19:37:58.550329] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-05-26 19:37:58.550381] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-05-26 19:37:58.550574] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.554767] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.558078] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.561376] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.564651] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.568012] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.571086] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.574171] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.577385] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.580527] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.583578] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport >[2014-05-26 19:37:58.587275] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-rep-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.37.62 > 5: option remote-subvolume /bricks/d1r1 > 6: option transport-type socket > 7: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 8: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-rep-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.37.215 > 16: option remote-subvolume /bricks/d1r2 > 17: option transport-type socket > 18: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 19: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-rep-replicate-0 > 24: type cluster/replicate > 25: subvolumes dist-rep-client-0 dist-rep-client-1 > 26: end-volume > 27: > 28: volume dist-rep-client-2 > 29: type protocol/client > 30: option ping-timeout 42 > 31: option remote-host 10.70.37.44 > 32: option remote-subvolume /bricks/d2r1 > 33: option transport-type socket > 34: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 35: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 36: option send-gids true > 37: end-volume > 38: > 39: volume dist-rep-client-3 > 40: type protocol/client > 41: option ping-timeout 42 > 42: option remote-host 10.70.37.201 > 43: option remote-subvolume /bricks/d2r2 > 44: option transport-type socket > 45: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 46: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 47: option send-gids true > 48: end-volume > 49: > 50: volume dist-rep-replicate-1 > 51: type cluster/replicate > 52: subvolumes dist-rep-client-2 dist-rep-client-3 > 53: end-volume > 54: > 55: volume dist-rep-client-4 > 56: type protocol/client > 57: option ping-timeout 42 > 58: option remote-host 10.70.37.62 > 59: option remote-subvolume /bricks/d3r1 > 60: option transport-type socket > 61: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 62: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 63: option send-gids true > 64: end-volume > 65: > 66: volume dist-rep-client-5 > 67: type protocol/client > 68: option ping-timeout 42 > 69: option remote-host 10.70.37.215 > 70: option remote-subvolume /bricks/d3r2 > 71: option transport-type socket > 72: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 73: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 74: option send-gids true > 75: end-volume > 76: > 77: volume dist-rep-replicate-2 > 78: type cluster/replicate > 79: subvolumes dist-rep-client-4 dist-rep-client-5 > 80: end-volume > 81: > 82: volume dist-rep-client-6 > 83: type protocol/client > 84: option ping-timeout 42 > 85: option remote-host 10.70.37.44 > 86: option remote-subvolume /bricks/d4r1 > 87: option transport-type socket > 88: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 89: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 90: option send-gids true > 91: end-volume > 92: > 93: volume dist-rep-client-7 > 94: type protocol/client > 95: option ping-timeout 42 > 96: option remote-host 10.70.37.201 > 97: option remote-subvolume /bricks/d4r2 > 98: option transport-type socket > 99: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >100: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >101: option send-gids true >102: end-volume >103: >104: volume dist-rep-replicate-3 >105: type cluster/replicate >106: subvolumes dist-rep-client-6 dist-rep-client-7 >107: end-volume >108: >109: volume dist-rep-client-8 >110: type protocol/client >111: option ping-timeout 42 >112: option remote-host 10.70.37.62 >113: option remote-subvolume /bricks/d5r1 >114: option transport-type socket >115: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >116: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >117: option send-gids true >118: end-volume >119: >120: volume dist-rep-client-9 >121: type protocol/client >122: option ping-timeout 42 >123: option remote-host 10.70.37.215 >124: option remote-subvolume /bricks/d5r2 >125: option transport-type socket >126: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >127: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >128: option send-gids true >129: end-volume >130: >131: volume dist-rep-replicate-4 >132: type cluster/replicate >133: subvolumes dist-rep-client-8 dist-rep-client-9 >134: end-volume >135: >136: volume dist-rep-client-10 >137: type protocol/client >138: option ping-timeout 42 >139: option remote-host 10.70.37.44 >140: option remote-subvolume /bricks/d6r1 >141: option transport-type socket >142: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >143: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >144: option send-gids true >145: end-volume >146: >147: volume dist-rep-client-11 >148: type protocol/client >149: option ping-timeout 42 >150: option remote-host 10.70.37.201 >151: option remote-subvolume /bricks/d6r2 >152: option transport-type socket >153: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >154: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >155: option send-gids true >156: end-volume >157: >158: volume dist-rep-replicate-5 >159: type cluster/replicate >160: subvolumes dist-rep-client-10 dist-rep-client-11 >161: end-volume >162: >163: volume dist-rep-dht >164: type cluster/distribute >165: subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5 >166: end-volume >167: >168: volume dist-rep-write-behind >169: type performance/write-behind >170: subvolumes dist-rep-dht >171: end-volume >172: >173: volume dist-rep >174: type debug/io-stats >175: option latency-measurement off >176: option count-fop-hits off >177: subvolumes dist-rep-write-behind >178: end-volume >179: >180: volume nfs-server >181: type nfs/server >182: option rpc-auth.auth-glusterfs on >183: option rpc-auth.auth-unix on >184: option rpc-auth.auth-null on >185: option rpc-auth.ports.insecure on >186: option rpc-auth-allow-insecure on >187: option transport-type socket >188: option transport.socket.listen-port 2049 >189: option nfs.dynamic-volumes on >190: option nfs.nlm on >191: option nfs.drc on >192: option rpc-auth.addr.dist-rep.allow * >193: option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5 >194: subvolumes dist-rep >195: end-volume >196: >+------------------------------------------------------------------------------+ >[2014-05-26 19:37:58.592844] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0) >[2014-05-26 19:37:58.593007] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0) >[2014-05-26 19:37:58.593079] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0) >[2014-05-26 19:37:58.606127] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.606240] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.606319] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.606720] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'. >[2014-05-26 19:37:58.606743] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.606828] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online. >[2014-05-26 19:37:58.606870] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'. >[2014-05-26 19:37:58.606881] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.606970] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online. >[2014-05-26 19:37:58.607000] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1 >[2014-05-26 19:37:58.607022] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'. >[2014-05-26 19:37:58.607031] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.607122] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online. >[2014-05-26 19:37:58.607167] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1 >[2014-05-26 19:37:58.607350] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1 >[2014-05-26 19:37:58.630217] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0) >[2014-05-26 19:37:58.636469] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0) >[2014-05-26 19:37:58.636539] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0) >[2014-05-26 19:37:58.648113] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.648635] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'. >[2014-05-26 19:37:58.648660] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.648761] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online. >[2014-05-26 19:37:58.649111] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0) >[2014-05-26 19:37:58.649173] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0) >[2014-05-26 19:37:58.649229] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0) >[2014-05-26 19:37:58.649287] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1 >[2014-05-26 19:37:58.664693] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.664804] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0) >[2014-05-26 19:37:58.664904] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0) >[2014-05-26 19:37:58.664985] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0) >[2014-05-26 19:37:58.665048] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.665289] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'. >[2014-05-26 19:37:58.665334] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.665483] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-6' came back up; going online. >[2014-05-26 19:37:58.678107] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1 >[2014-05-26 19:37:58.678143] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.678231] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.678333] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'. >[2014-05-26 19:37:58.678349] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.678399] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online. >[2014-05-26 19:37:58.678588] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'. >[2014-05-26 19:37:58.678605] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.679070] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1 >[2014-05-26 19:37:58.679098] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.679167] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.679298] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'. >[2014-05-26 19:37:58.679313] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.679490] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'. >[2014-05-26 19:37:58.679520] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.679690] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.679771] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1 >[2014-05-26 19:37:58.679838] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'. >[2014-05-26 19:37:58.679852] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.680037] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1 >[2014-05-26 19:37:58.680088] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1 >[2014-05-26 19:37:58.680108] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:37:58.680476] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1 >[2014-05-26 19:37:58.680796] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'. >[2014-05-26 19:37:58.680811] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.681098] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1 >[2014-05-26 19:37:58.681913] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'. >[2014-05-26 19:37:58.681930] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:37:58.682402] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1 >[2014-05-26 19:42:04.841386] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available) >[2014-05-26 19:42:12.527990] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-26 19:42:13.542756] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-26 19:42:14.566862] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-05-26 19:42:14.628849] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON >[2014-05-26 19:42:14.634054] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-05-26 19:42:14.634405] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-26 19:42:14.645448] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-05-26 19:42:14.645547] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-05-26 19:42:14.645598] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-05-26 19:42:14.645771] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.649428] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.652819] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.655856] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.658814] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.661786] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.664760] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.667702] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.670662] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.673640] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.676613] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport >[2014-05-26 19:42:14.679558] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-rep-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.37.62 > 5: option remote-subvolume /bricks/d1r1 > 6: option transport-type socket > 7: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 8: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-rep-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.37.215 > 16: option remote-subvolume /bricks/d1r2 > 17: option transport-type socket > 18: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 19: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-rep-replicate-0 > 24: type cluster/replicate > 25: subvolumes dist-rep-client-0 dist-rep-client-1 > 26: end-volume > 27: > 28: volume dist-rep-client-2 > 29: type protocol/client > 30: option ping-timeout 42 > 31: option remote-host 10.70.37.44 > 32: option remote-subvolume /bricks/d2r1 > 33: option transport-type socket > 34: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 35: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 36: option send-gids true > 37: end-volume > 38: > 39: volume dist-rep-client-3 > 40: type protocol/client > 41: option ping-timeout 42 > 42: option remote-host 10.70.37.201 > 43: option remote-subvolume /bricks/d2r2 > 44: option transport-type socket > 45: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 46: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 47: option send-gids true > 48: end-volume > 49: > 50: volume dist-rep-replicate-1 > 51: type cluster/replicate > 52: subvolumes dist-rep-client-2 dist-rep-client-3 > 53: end-volume > 54: > 55: volume dist-rep-client-4 > 56: type protocol/client > 57: option ping-timeout 42 > 58: option remote-host 10.70.37.62 > 59: option remote-subvolume /bricks/d3r1 > 60: option transport-type socket > 61: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 62: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 63: option send-gids true > 64: end-volume > 65: > 66: volume dist-rep-client-5 > 67: type protocol/client > 68: option ping-timeout 42 > 69: option remote-host 10.70.37.215 > 70: option remote-subvolume /bricks/d3r2 > 71: option transport-type socket > 72: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 73: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 74: option send-gids true > 75: end-volume > 76: > 77: volume dist-rep-replicate-2 > 78: type cluster/replicate > 79: subvolumes dist-rep-client-4 dist-rep-client-5 > 80: end-volume > 81: > 82: volume dist-rep-client-6 > 83: type protocol/client > 84: option ping-timeout 42 > 85: option remote-host 10.70.37.44 > 86: option remote-subvolume /bricks/d4r1 > 87: option transport-type socket > 88: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 89: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 90: option send-gids true > 91: end-volume > 92: > 93: volume dist-rep-client-7 > 94: type protocol/client > 95: option ping-timeout 42 > 96: option remote-host 10.70.37.201 > 97: option remote-subvolume /bricks/d4r2 > 98: option transport-type socket > 99: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >100: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >101: option send-gids true >102: end-volume >103: >104: volume dist-rep-replicate-3 >105: type cluster/replicate >106: subvolumes dist-rep-client-6 dist-rep-client-7 >107: end-volume >108: >109: volume dist-rep-client-8 >110: type protocol/client >111: option ping-timeout 42 >112: option remote-host 10.70.37.62 >113: option remote-subvolume /bricks/d5r1 >114: option transport-type socket >115: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >116: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >117: option send-gids true >118: end-volume >119: >120: volume dist-rep-client-9 >121: type protocol/client >122: option ping-timeout 42 >123: option remote-host 10.70.37.215 >124: option remote-subvolume /bricks/d5r2 >125: option transport-type socket >126: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >127: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >128: option send-gids true >129: end-volume >130: >131: volume dist-rep-replicate-4 >132: type cluster/replicate >133: subvolumes dist-rep-client-8 dist-rep-client-9 >134: end-volume >135: >136: volume dist-rep-client-10 >137: type protocol/client >138: option ping-timeout 42 >139: option remote-host 10.70.37.44 >140: option remote-subvolume /bricks/d6r1 >141: option transport-type socket >142: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >143: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >144: option send-gids true >145: end-volume >146: >147: volume dist-rep-client-11 >148: type protocol/client >149: option ping-timeout 42 >150: option remote-host 10.70.37.201 >151: option remote-subvolume /bricks/d6r2 >152: option transport-type socket >153: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >154: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >155: option send-gids true >156: end-volume >157: >158: volume dist-rep-replicate-5 >159: type cluster/replicate >160: subvolumes dist-rep-client-10 dist-rep-client-11 >161: end-volume >162: >163: volume dist-rep-dht >164: type cluster/distribute >165: subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5 >166: end-volume >167: >168: volume dist-rep-write-behind >169: type performance/write-behind >170: subvolumes dist-rep-dht >171: end-volume >172: >173: volume dist-rep >174: type debug/io-stats >175: option latency-measurement off >176: option count-fop-hits off >177: subvolumes dist-rep-write-behind >178: end-volume >179: >180: volume nfs-server >181: type nfs/server >182: option rpc-auth.auth-glusterfs on >183: option rpc-auth.auth-unix on >184: option rpc-auth.auth-null on >185: option rpc-auth.ports.insecure on >186: option rpc-auth-allow-insecure on >187: option transport-type socket >188: option transport.socket.listen-port 2049 >189: option nfs.dynamic-volumes on >190: option nfs.nlm on >191: option nfs.drc on >192: option rpc-auth.addr.dist-rep.allow * >193: option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5 >194: subvolumes dist-rep >195: end-volume >196: >+------------------------------------------------------------------------------+ >[2014-05-26 19:42:14.684290] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:14.684372] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:14.684399] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-4: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:14.684445] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-8: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:14.684477] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:14.684494] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:14.684890] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0) >[2014-05-26 19:42:14.688058] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0) >[2014-05-26 19:42:14.688137] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0) >[2014-05-26 19:42:14.688233] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0) >[2014-05-26 19:42:14.697283] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0) >[2014-05-26 19:42:14.697331] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0) >[2014-05-26 19:42:14.703493] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.703564] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.703621] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.704141] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'. >[2014-05-26 19:42:14.704182] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.704238] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online. >[2014-05-26 19:42:14.704295] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'. >[2014-05-26 19:42:14.704308] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.704358] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-2' came back up; going online. >[2014-05-26 19:42:14.704608] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.704675] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1 >[2014-05-26 19:42:14.704863] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1 >[2014-05-26 19:42:14.704979] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.705175] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'. >[2014-05-26 19:42:14.705191] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.705247] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-6' came back up; going online. >[2014-05-26 19:42:14.705276] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:14.705587] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1 >[2014-05-26 19:42:14.705757] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'. >[2014-05-26 19:42:14.705772] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.705927] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'. >[2014-05-26 19:42:14.705942] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.706311] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1 >[2014-05-26 19:42:14.706572] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1 >[2014-05-26 19:42:14.706820] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'. >[2014-05-26 19:42:14.706842] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:14.707334] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1 >[2014-05-26 19:42:15.749343] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:15.749445] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-1: disconnected from dist-rep-client-1. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:15.749466] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up. >[2014-05-26 19:42:15.749580] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-5: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:15.749676] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-rep-client-9: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-05-26 19:42:15.749780] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-5: disconnected from dist-rep-client-5. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:15.749805] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up. >[2014-05-26 19:42:15.749831] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-9: disconnected from dist-rep-client-9. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 19:42:15.749856] E [MSGID: 108006] [afr-common.c:4164:afr_notify] 0-dist-rep-replicate-4: All subvolumes are down. Going offline until atleast one of them comes back up. >[2014-05-26 19:42:15.756188] I [dht-layout.c:653:dht_layout_normalize] 0-dist-rep-dht: found anomalies in /. holes=3 overlaps=0 >[2014-05-26 19:42:15.756271] W [dht-selfheal.c:952:dht_selfheal_directory] 0-dist-rep-dht: 3 subvolumes down -- not fixing >[2014-05-26 19:42:18.552152] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0) >[2014-05-26 19:42:18.561550] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:18.562012] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'. >[2014-05-26 19:42:18.562044] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:18.562327] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online. >[2014-05-26 19:42:18.562565] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1 >[2014-05-26 19:42:18.565790] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0) >[2014-05-26 19:42:18.578994] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:18.579311] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'. >[2014-05-26 19:42:18.579331] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:18.579403] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online. >[2014-05-26 19:42:18.579499] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1 >[2014-05-26 19:42:18.579572] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0) >[2014-05-26 19:42:18.586281] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:18.586727] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'. >[2014-05-26 19:42:18.586750] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:18.586829] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online. >[2014-05-26 19:42:18.587026] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1 >[2014-05-26 19:42:21.876433] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0) >[2014-05-26 19:42:21.882538] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:21.883038] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'. >[2014-05-26 19:42:21.883086] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:21.883338] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1 >[2014-05-26 19:42:24.593551] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0) >[2014-05-26 19:42:24.601229] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:24.601321] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0) >[2014-05-26 19:42:24.610727] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'. >[2014-05-26 19:42:24.610751] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:24.611067] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1 >[2014-05-26 19:42:24.611126] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 19:42:24.611536] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'. >[2014-05-26 19:42:24.611560] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 19:42:24.611812] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1 >[2014-05-26 19:46:26.001795] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:46:26.064259] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:46:26.064800] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:46:26.068462] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:56:26.793882] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:56:26.862455] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 19:56:26.865036] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 19:56:26.865135] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:12:34.579086] W [socket.c:522:__socket_rwv] 0-dist-rep-client-4: readv on 10.70.37.62:49154 failed (No data available) >[2014-05-26 20:12:34.579193] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-4: disconnected from dist-rep-client-4. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 20:12:34.579506] W [socket.c:522:__socket_rwv] 0-dist-rep-client-0: readv on 10.70.37.62:49153 failed (No data available) >[2014-05-26 20:12:34.579625] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-0: disconnected from dist-rep-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 20:12:34.579671] W [socket.c:522:__socket_rwv] 0-dist-rep-client-8: readv on 10.70.37.62:49155 failed (No data available) >[2014-05-26 20:12:34.579736] I [client.c:2215:client_rpc_notify] 0-dist-rep-client-8: disconnected from dist-rep-client-8. Client process will keep trying to connect to glusterd until brick's port is available >[2014-05-26 20:23:53.454547] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.7 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/d09990542e7861f6452f94f6f995eae3.socket) >[2014-05-26 20:23:53.474993] I [rpcsvc.c:2120:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-05-26 20:23:53.500618] I [rpc-drc.c:730:rpcsvc_drc_init] 0-rpc-service: DRC is turned ON >[2014-05-26 20:23:53.506231] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-05-26 20:23:53.506470] I [dht-shared.c:332:dht_init_regex] 0-dist-rep-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-05-26 20:23:53.516037] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-05-26 20:23:53.516129] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-05-26 20:23:53.516185] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-05-26 20:23:53.516357] I [client.c:2280:notify] 0-dist-rep-client-0: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.519520] I [client.c:2280:notify] 0-dist-rep-client-1: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.522553] I [client.c:2280:notify] 0-dist-rep-client-2: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.525575] I [client.c:2280:notify] 0-dist-rep-client-3: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.528613] I [client.c:2280:notify] 0-dist-rep-client-4: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.531904] I [client.c:2280:notify] 0-dist-rep-client-5: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.534942] I [client.c:2280:notify] 0-dist-rep-client-6: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.537935] I [client.c:2280:notify] 0-dist-rep-client-7: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.540906] I [client.c:2280:notify] 0-dist-rep-client-8: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.544041] I [client.c:2280:notify] 0-dist-rep-client-9: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.547111] I [client.c:2280:notify] 0-dist-rep-client-10: parent translators are ready, attempting connect on transport >[2014-05-26 20:23:53.550079] I [client.c:2280:notify] 0-dist-rep-client-11: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-rep-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.37.62 > 5: option remote-subvolume /bricks/d1r1 > 6: option transport-type socket > 7: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 8: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-rep-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.37.215 > 16: option remote-subvolume /bricks/d1r2 > 17: option transport-type socket > 18: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 19: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-rep-replicate-0 > 24: type cluster/replicate > 25: subvolumes dist-rep-client-0 dist-rep-client-1 > 26: end-volume > 27: > 28: volume dist-rep-client-2 > 29: type protocol/client > 30: option ping-timeout 42 > 31: option remote-host 10.70.37.44 > 32: option remote-subvolume /bricks/d2r1 > 33: option transport-type socket > 34: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 35: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 36: option send-gids true > 37: end-volume > 38: > 39: volume dist-rep-client-3 > 40: type protocol/client > 41: option ping-timeout 42 > 42: option remote-host 10.70.37.201 > 43: option remote-subvolume /bricks/d2r2 > 44: option transport-type socket > 45: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 46: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 47: option send-gids true > 48: end-volume > 49: > 50: volume dist-rep-replicate-1 > 51: type cluster/replicate > 52: subvolumes dist-rep-client-2 dist-rep-client-3 > 53: end-volume > 54: > 55: volume dist-rep-client-4 > 56: type protocol/client > 57: option ping-timeout 42 > 58: option remote-host 10.70.37.62 > 59: option remote-subvolume /bricks/d3r1 > 60: option transport-type socket > 61: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 62: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 63: option send-gids true > 64: end-volume > 65: > 66: volume dist-rep-client-5 > 67: type protocol/client > 68: option ping-timeout 42 > 69: option remote-host 10.70.37.215 > 70: option remote-subvolume /bricks/d3r2 > 71: option transport-type socket > 72: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 73: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 74: option send-gids true > 75: end-volume > 76: > 77: volume dist-rep-replicate-2 > 78: type cluster/replicate > 79: subvolumes dist-rep-client-4 dist-rep-client-5 > 80: end-volume > 81: > 82: volume dist-rep-client-6 > 83: type protocol/client > 84: option ping-timeout 42 > 85: option remote-host 10.70.37.44 > 86: option remote-subvolume /bricks/d4r1 > 87: option transport-type socket > 88: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 > 89: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 > 90: option send-gids true > 91: end-volume > 92: > 93: volume dist-rep-client-7 > 94: type protocol/client > 95: option ping-timeout 42 > 96: option remote-host 10.70.37.201 > 97: option remote-subvolume /bricks/d4r2 > 98: option transport-type socket > 99: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >100: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >101: option send-gids true >102: end-volume >103: >104: volume dist-rep-replicate-3 >105: type cluster/replicate >106: subvolumes dist-rep-client-6 dist-rep-client-7 >107: end-volume >108: >109: volume dist-rep-client-8 >110: type protocol/client >111: option ping-timeout 42 >112: option remote-host 10.70.37.62 >113: option remote-subvolume /bricks/d5r1 >114: option transport-type socket >115: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >116: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >117: option send-gids true >118: end-volume >119: >120: volume dist-rep-client-9 >121: type protocol/client >122: option ping-timeout 42 >123: option remote-host 10.70.37.215 >124: option remote-subvolume /bricks/d5r2 >125: option transport-type socket >126: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >127: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >128: option send-gids true >129: end-volume >130: >131: volume dist-rep-replicate-4 >132: type cluster/replicate >133: subvolumes dist-rep-client-8 dist-rep-client-9 >134: end-volume >135: >136: volume dist-rep-client-10 >137: type protocol/client >138: option ping-timeout 42 >139: option remote-host 10.70.37.44 >140: option remote-subvolume /bricks/d6r1 >141: option transport-type socket >142: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >143: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >144: option send-gids true >145: end-volume >146: >147: volume dist-rep-client-11 >148: type protocol/client >149: option ping-timeout 42 >150: option remote-host 10.70.37.201 >151: option remote-subvolume /bricks/d6r2 >152: option transport-type socket >153: option username a07d99e5-0ecc-4f52-a886-35cd82e94dd9 >154: option password 29ee8f8d-f6cd-4d8f-bf21-b2d11281f770 >155: option send-gids true >156: end-volume >157: >158: volume dist-rep-replicate-5 >159: type cluster/replicate >160: subvolumes dist-rep-client-10 dist-rep-client-11 >161: end-volume >162: >163: volume dist-rep-dht >164: type cluster/distribute >165: subvolumes dist-rep-replicate-0 dist-rep-replicate-1 dist-rep-replicate-2 dist-rep-replicate-3 dist-rep-replicate-4 dist-rep-replicate-5 >166: end-volume >167: >168: volume dist-rep-write-behind >169: type performance/write-behind >170: subvolumes dist-rep-dht >171: end-volume >172: >173: volume dist-rep >174: type debug/io-stats >175: option latency-measurement off >176: option count-fop-hits off >177: subvolumes dist-rep-write-behind >178: end-volume >179: >180: volume nfs-server >181: type nfs/server >182: option rpc-auth.auth-glusterfs on >183: option rpc-auth.auth-unix on >184: option rpc-auth.auth-null on >185: option rpc-auth.ports.insecure on >186: option rpc-auth-allow-insecure on >187: option transport-type socket >188: option transport.socket.listen-port 2049 >189: option nfs.dynamic-volumes on >190: option nfs.nlm on >191: option nfs.drc on >192: option rpc-auth.addr.dist-rep.allow * >193: option nfs3.dist-rep.volume-id 3ddfa9e6-481f-4a41-b0ab-9b6b762ddea5 >194: subvolumes dist-rep >195: end-volume >196: >+------------------------------------------------------------------------------+ >[2014-05-26 20:23:53.555077] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-0: changing port to 49153 (from 0) >[2014-05-26 20:23:53.558365] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-4: changing port to 49154 (from 0) >[2014-05-26 20:23:53.558420] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-8: changing port to 49155 (from 0) >[2014-05-26 20:23:53.565130] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.565228] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.565316] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.565406] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-4: Connected to dist-rep-client-4, attached to remote volume '/bricks/d3r1'. >[2014-05-26 20:23:53.565428] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.565518] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-2: Subvolume 'dist-rep-client-4' came back up; going online. >[2014-05-26 20:23:53.565612] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-4: Server lk version = 1 >[2014-05-26 20:23:53.568313] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-8: Connected to dist-rep-client-8, attached to remote volume '/bricks/d5r1'. >[2014-05-26 20:23:53.568331] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.568376] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-4: Subvolume 'dist-rep-client-8' came back up; going online. >[2014-05-26 20:23:53.568497] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-8: Server lk version = 1 >[2014-05-26 20:23:53.568655] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-0: Connected to dist-rep-client-0, attached to remote volume '/bricks/d1r1'. >[2014-05-26 20:23:53.568671] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.568712] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-0: Subvolume 'dist-rep-client-0' came back up; going online. >[2014-05-26 20:23:53.568859] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-0: Server lk version = 1 >[2014-05-26 20:23:53.594410] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-3: changing port to 49158 (from 0) >[2014-05-26 20:23:53.600593] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-7: changing port to 49159 (from 0) >[2014-05-26 20:23:53.600820] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-1: changing port to 49155 (from 0) >[2014-05-26 20:23:53.600920] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-5: changing port to 49156 (from 0) >[2014-05-26 20:23:53.601025] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-9: changing port to 49157 (from 0) >[2014-05-26 20:23:53.615483] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-11: changing port to 49160 (from 0) >[2014-05-26 20:23:53.615577] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-2: changing port to 49158 (from 0) >[2014-05-26 20:23:53.615672] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-6: changing port to 49159 (from 0) >[2014-05-26 20:23:53.615722] I [rpc-clnt.c:1763:rpc_clnt_reconfig] 0-dist-rep-client-10: changing port to 49160 (from 0) >[2014-05-26 20:23:53.628243] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.628342] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.628478] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.628570] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.628798] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.628884] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-1: Connected to dist-rep-client-1, attached to remote volume '/bricks/d1r2'. >[2014-05-26 20:23:53.628899] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.629173] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-3: Connected to dist-rep-client-3, attached to remote volume '/bricks/d2r2'. >[2014-05-26 20:23:53.629191] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.629251] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-1: Subvolume 'dist-rep-client-3' came back up; going online. >[2014-05-26 20:23:53.629353] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-5: Connected to dist-rep-client-5, attached to remote volume '/bricks/d3r2'. >[2014-05-26 20:23:53.629368] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.629709] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.629796] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-7: Connected to dist-rep-client-7, attached to remote volume '/bricks/d4r2'. >[2014-05-26 20:23:53.629819] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.629894] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-3: Subvolume 'dist-rep-client-7' came back up; going online. >[2014-05-26 20:23:53.629934] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.630015] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-5: Server lk version = 1 >[2014-05-26 20:23:53.630060] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.630120] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-3: Server lk version = 1 >[2014-05-26 20:23:53.630163] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-rep-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-05-26 20:23:53.630215] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-1: Server lk version = 1 >[2014-05-26 20:23:53.630327] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-7: Server lk version = 1 >[2014-05-26 20:23:53.630651] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-2: Connected to dist-rep-client-2, attached to remote volume '/bricks/d2r1'. >[2014-05-26 20:23:53.630667] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.631177] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-6: Connected to dist-rep-client-6, attached to remote volume '/bricks/d4r1'. >[2014-05-26 20:23:53.631200] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.631354] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-2: Server lk version = 1 >[2014-05-26 20:23:53.632189] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-6: Server lk version = 1 >[2014-05-26 20:23:53.632247] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-9: Connected to dist-rep-client-9, attached to remote volume '/bricks/d5r2'. >[2014-05-26 20:23:53.632261] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.632546] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-9: Server lk version = 1 >[2014-05-26 20:23:53.633622] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-10: Connected to dist-rep-client-10, attached to remote volume '/bricks/d6r1'. >[2014-05-26 20:23:53.633642] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.633690] I [MSGID: 108005] [afr-common.c:4126:afr_notify] 0-dist-rep-replicate-5: Subvolume 'dist-rep-client-10' came back up; going online. >[2014-05-26 20:23:53.634043] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-10: Server lk version = 1 >[2014-05-26 20:23:53.634244] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-rep-client-11: Connected to dist-rep-client-11, attached to remote volume '/bricks/d6r2'. >[2014-05-26 20:23:53.634259] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-rep-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-05-26 20:23:53.634698] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-rep-client-11: Server lk version = 1 >[2014-05-26 20:45:30.499777] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:45:30.500682] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:47:36.560612] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:47:36.561694] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:47:48.296437] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:47:48.298327] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:48:26.647506] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:48:26.712162] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:48:26.716245] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:48:26.716990] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:48:46.056673] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:48:46.121892] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:48:46.126373] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:48:46.126433] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:49:25.908758] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:49:25.909521] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:58:47.058336] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:58:47.059406] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:58:53.439086] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:58:53.509725] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:58:53.514085] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:58:53.514178] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:58:56.658559] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:58:56.720248] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:58:56.726084] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:58:56.726122] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:59:03.738805] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:59:03.739852] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:59:08.975778] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-4: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:59:08.976858] W [client-rpc-fops.c:1088:client3_3_getxattr_cbk] 0-dist-rep-client-5: remote operation failed: Permission denied. Path: <gfid:c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd> (c3a3e8c5-0654-4de9-acdf-1a3562d4e1fd). Key: (null) >[2014-05-26 20:59:24.798614] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:59:24.860331] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:59:24.863680] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:59:24.864117] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:59:46.004545] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:59:46.070053] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-05-26 20:59:46.074438] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-05-26 20:59:46.074666] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1101438
: 899401 |
954119
|
954361