Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 929196 Details for
Bug 1132489
DHT: kernel untar fails on the NFS mount after add-brick
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
nfs logs
nfs.log (text/x-log), 728.81 KB, created by
shylesh
on 2014-08-21 12:38:56 UTC
(
hide
)
Description:
nfs logs
Filename:
MIME Type:
Creator:
shylesh
Created:
2014-08-21 12:38:56 UTC
Size:
728.81 KB
patch
obsolete
>[2014-08-16 21:37:02.576434] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-18 08:49:46.123686] W [rpcsvc.c:261:rpcsvc_program_actor] 0-rpc-service: RPC program version not available (req 100003 4) >[2014-08-18 08:49:46.123737] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >[2014-08-18 08:55:30.057935] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-0: remote operation failed: File exists. Path: /b2 >[2014-08-18 08:55:30.060889] W [client-rpc-fops.c:2539:client3_3_rename_cbk] 0-test0-client-1: remote operation failed: No such file or directory >[2014-08-18 08:55:30.060923] W [MSGID: 109030] [dht-rename.c:587:dht_rename_cbk] 0-test0-dht: /a2: Rename on test0-client-1 failed, (gfid = 3d8c8a65-c5ae-4b55-b10c-5051b4b9d793) [No such file or directory] >[2014-08-18 08:55:30.060949] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: 933b4092: rename /a2 -> /b2 => -1 (No such file or directory) >[2014-08-18 08:58:54.296368] W [client-rpc-fops.c:2539:client3_3_rename_cbk] 0-test0-client-1: remote operation failed: No such file or directory >[2014-08-18 08:58:54.296450] W [MSGID: 109030] [dht-rename.c:587:dht_rename_cbk] 0-test0-dht: /b5: Rename on test0-client-1 failed, (gfid = f4d80045-2cd0-4add-b4fe-a08d5d7101fe) [No such file or directory] >[2014-08-18 08:58:54.296907] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: d93b4092: rename /b5 -> /c5 => -1 (No such file or directory) >[2014-08-18 08:58:54.305295] W [client-rpc-fops.c:2539:client3_3_rename_cbk] 0-test0-client-1: remote operation failed: No such file or directory >[2014-08-18 08:58:54.305377] W [MSGID: 109030] [dht-rename.c:587:dht_rename_cbk] 0-test0-dht: /b6: Rename on test0-client-1 failed, (gfid = e30af3c7-60dd-4a7c-a283-c28f950a7478) [No such file or directory] >[2014-08-18 08:58:54.305893] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: dc3b4092: rename /b6 -> /c6 => -1 (No such file or directory) >[2014-08-18 08:58:54.313357] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /b7: Rename: unlink on test0-client-1 failed [No such file or directory] >[2014-08-18 08:58:54.340816] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /b10: Rename: unlink on test0-client-1 failed [No such file or directory] >[2014-08-18 09:01:03.970978] W [client-rpc-fops.c:2539:client3_3_rename_cbk] 0-test0-client-1: remote operation failed: No such file or directory >[2014-08-18 09:01:03.971015] W [MSGID: 109030] [dht-rename.c:587:dht_rename_cbk] 0-test0-dht: /d5: Rename on test0-client-1 failed, (gfid = f4d80045-2cd0-4add-b4fe-a08d5d7101fe) [No such file or directory] >[2014-08-18 09:01:03.971591] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: 653c4092: rename /d5 -> /e5 => -1 (No such file or directory) >[2014-08-18 09:01:04.014833] W [client-rpc-fops.c:2607:client3_3_link_cbk] 0-test0-client-2: remote operation failed: No such file or directory (/d10 -> /e10) >[2014-08-18 09:01:04.015480] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /d10: Rename: unlink on test0-client-2 failed [No such file or directory] >[2014-08-18 09:01:04.015515] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: 7b3c4092: rename /d10 -> /e10 => -1 (Structure needs cleaning) >[2014-08-18 09:01:04.015535] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 7b3c4092, RENAME: NFS: 10006(Error occurred on the server or IO Error), POSIX: 14(Bad address) >[2014-08-18 10:24:36.722628] W [socket.c:529:__socket_rwv] 0-test0-client-0: readv on 10.70.35.198:49187 failed (No data available) >[2014-08-18 10:24:36.722754] I [client.c:2215:client_rpc_notify] 0-test0-client-0: disconnected from test0-client-0. Client process will keep trying to connect to glusterd until brick's port is available >[2014-08-18 10:24:47.410822] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:24:47.415109] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:24:51.415890] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:24:52.419217] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:24:55.422059] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:24:55.426401] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:24:59.427219] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:24:59.431624] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:03.432169] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:03.436676] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:07.440698] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:07.445078] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:11.442533] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:11.446896] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:15.447713] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:15.452547] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:19.452961] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:19.457509] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:23.458046] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:23.462398] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:27.463252] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:27.467713] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:31.468441] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:31.472820] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:35.473633] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:35.478036] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:39.478825] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:39.483412] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:43.484010] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:43.488444] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:47.489241] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:47.493563] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:51.494235] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:51.498585] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:55.499281] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:55.503596] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:25:59.504327] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:25:59.508833] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:03.509341] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:03.513920] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:07.514493] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:07.518820] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:11.519451] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:11.523816] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:15.524426] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:15.529062] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:19.529561] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:19.533939] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:23.534459] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:23.538778] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:27.539463] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:27.543796] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:31.544610] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:31.548938] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:35.549742] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:35.554072] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:39.554804] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:39.559622] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:43.559914] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:43.564289] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:47.565034] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:47.569373] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:51.570092] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:51.574434] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:55.575169] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:55.579571] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:26:59.580167] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:26:59.584518] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:03.585233] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:03.589559] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:07.590286] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:07.594599] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:11.595353] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:11.599678] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:15.600445] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:15.605030] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:19.605731] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:19.610046] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:23.610681] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:23.615001] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:27.615727] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:27.620049] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:31.620788] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:31.625106] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:35.625804] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:35.630127] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:39.630865] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:39.635245] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:43.635878] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:43.640240] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:47.640952] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:47.645296] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:51.646043] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:51.650372] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:55.651082] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:55.655478] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:27:59.656269] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:27:59.660578] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:03.661221] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:03.665882] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:07.666386] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:07.670726] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:11.671517] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:11.675840] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:13.976949] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f1 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.978593] W [client-rpc-fops.c:503:client3_3_stat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:13.980437] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e1 (94e8046d-6520-486f-8092-189c73a52366) >[2014-08-18 10:28:13.980482] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 8d3c4092: /e1 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:13.980583] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e1 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.980619] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e1 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.981741] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e1 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.981792] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e1 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.986885] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f2 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.988075] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e2 (3d8c8a65-c5ae-4b55-b10c-5051b4b9d793) >[2014-08-18 10:28:13.988104] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 913c4092: /e2 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:13.988161] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e2 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.988187] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e2 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.989609] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e2 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.989642] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e2 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.994500] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f3 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.995691] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e3 (cf9d0b9a-c742-461a-b629-22d40ae8e35d) >[2014-08-18 10:28:13.995720] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 953c4092: /e3 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:13.996303] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e3 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.996390] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e3 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:13.996428] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 953c4092: /e3 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:13.996465] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 953c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:13.997879] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e3 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:13.997904] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e3 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:13.997919] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 963c4092: /e3 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:13.997940] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 963c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:14.001535] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f4 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.001600] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f4 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.003162] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e4 (a874dac0-a8fd-4ab1-af2b-4cb160f011d7) >[2014-08-18 10:28:14.003207] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 993c4092: /e4 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:14.003964] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e4 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.003991] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e4 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:14.004007] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 993c4092: /e4 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:14.004028] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 993c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:14.005000] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e4 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.005028] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e4 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:14.005052] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: 9a3c4092: /e4 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:14.005073] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 9a3c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:14.009022] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f5 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.010709] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e5 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.013994] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f5 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:14.015585] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-1: remote operation failed: File exists. Path: /f5 >[2014-08-18 10:28:14.016258] W [dht-linkfile.c:49:dht_linkfile_lookup_cbk] 0-test0-dht: got non-linkfile test0-client-1:/f5, gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:15.676757] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:15.681105] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:19.681734] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:19.686060] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:21.918423] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.918491] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.919371] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.920023] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f6 >[2014-08-18 10:28:21.921628] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e6 (e30af3c7-60dd-4a7c-a283-c28f950a7478) >[2014-08-18 10:28:21.922336] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.922371] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.923539] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.923574] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e6 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.928155] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.928193] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.928773] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.929392] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f7 >[2014-08-18 10:28:21.930248] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e7 (ef656ac7-2fb6-41af-a579-6b8e3ac61c14) >[2014-08-18 10:28:21.930882] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.930914] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.931828] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.931862] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e7 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.935533] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.935569] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.938567] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.938601] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.940394] W [client-rpc-fops.c:503:client3_3_stat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:21.942870] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.942905] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.943911] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.943944] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f8 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.944548] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e8 >[2014-08-18 10:28:21.945447] W [nfs3.c:3673:nfs3svc_rename_cbk] 0-nfs: b33c4092: rename /e8 -> /f8 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:21.945482] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: b33c4092, RENAME: NFS: 5(I/O error), POSIX: 14(Bad address) >[2014-08-18 10:28:21.948555] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f9 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.948590] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f9 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.949540] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e9 (95c026db-9868-4ac9-b19b-9ca61ffe55c3) >[2014-08-18 10:28:21.949568] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: b63c4092: /e9 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:21.950210] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e9 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.950236] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e9 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:21.950252] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: b63c4092: /e9 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:21.950295] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: b63c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:21.951160] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e9 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.951186] I [dht-common.c:1433:dht_lookup_linkfile_cbk] 0-test0-dht: Lookup of /e9 on test0-client-0 (following linkfile) failed (Transport endpoint is not connected),gfid = 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:21.951202] W [nfs3.c:1230:nfs3svc_lookup_cbk] 0-nfs: b73c4092: /e9 => -1 (Transport endpoint is not connected) >[2014-08-18 10:28:21.951222] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: b73c4092, LOOKUP: NFS: 5(I/O error), POSIX: 107(Transport endpoint is not connected), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 >[2014-08-18 10:28:21.954785] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /f10 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.955777] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e10 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:21.955811] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: /e10 (00000000-0000-0000-0000-000000000000) >[2014-08-18 10:28:23.686861] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:23.691177] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:27.691873] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:27.696397] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:31.697069] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:31.701449] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:35.702037] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:35.706395] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:39.707066] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:39.711427] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:43.712010] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:43.716408] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:46.825254] W [client-rpc-fops.c:503:client3_3_stat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:46.827277] W [client-rpc-fops.c:503:client3_3_stat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:46.828992] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected. Path: / (00000000-0000-0000-0000-000000000001) >[2014-08-18 10:28:46.829270] W [client-rpc-fops.c:2470:client3_3_readdirp_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:46.830471] W [client-rpc-fops.c:1467:client3_3_fstat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:46.834273] W [client-rpc-fops.c:1467:client3_3_fstat_cbk] 0-test0-client-0: remote operation failed: Transport endpoint is not connected >[2014-08-18 10:28:47.717133] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:47.721475] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:51.722003] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:51.726330] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:55.726983] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:55.731517] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:28:59.732029] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:28:59.736530] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:03.737105] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:03.741621] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:07.742253] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:07.746614] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:11.747499] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:11.751811] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:15.752478] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:15.756799] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:19.757682] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:19.761999] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:23.762740] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:23.767231] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:27.767876] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:27.772379] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:31.772982] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:31.777509] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:35.778164] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:35.782513] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:39.783208] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:39.787538] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:43.788461] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:43.792834] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:47.793493] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:47.797839] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:51.798744] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:51.803073] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:55.803977] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:55.808359] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:29:59.808986] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:29:59.813333] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:03.814140] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:03.818526] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:07.819462] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:07.823850] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:11.824514] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:11.828872] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:15.829651] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:15.834021] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:19.834607] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:19.838999] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:23.839691] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:23.844022] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:27.844714] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:27.849051] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:31.849912] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:31.854320] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:35.855027] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:35.859378] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:39.859971] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:39.864480] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:43.865055] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:43.869574] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:47.870463] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:47.874915] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:51.875683] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:51.880118] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:55.880906] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:55.885382] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:30:59.886132] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:30:59.890486] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:03.891241] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:03.895614] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:07.896379] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:07.900700] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:11.901453] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:11.906036] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:15.906608] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:15.910935] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:19.911870] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:19.916215] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:23.917008] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:23.921348] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:27.922030] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:27.926540] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:31.927175] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:31.931523] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:35.932152] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:35.936741] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:39.937438] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:39.941772] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:43.942454] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:43.946950] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:47.947734] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:47.952046] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:51.952879] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:51.957211] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:55.958025] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:55.962362] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:31:59.963118] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:31:59.967466] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:03.968235] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:03.972559] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:07.973336] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:07.977862] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:11.978613] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:11.982933] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:15.983744] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:15.988305] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:19.988786] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:19.993288] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:23.993969] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:23.998550] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:27.999200] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:28.003525] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:32.004536] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:32.008907] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:36.009683] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:36.014025] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:40.014655] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:40.019002] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:44.019729] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:44.024056] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:48.024778] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:48.029148] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:52.029826] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:52.034134] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:32:56.034971] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:32:56.039301] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:00.040047] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:00.044388] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:04.045115] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:04.049628] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:08.050296] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:08.054622] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:12.055378] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:12.059719] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:16.060522] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:16.064845] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:20.065516] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:20.069838] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:24.070708] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:24.075173] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:28.076087] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:28.080483] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:32.081078] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:32.085430] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:36.086225] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:36.090611] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:40.091319] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:40.095661] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:44.096408] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:44.100750] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:48.101322] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:48.105666] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:52.106312] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:52.110798] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:33:56.111281] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:33:56.115730] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:00.116420] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:00.120731] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:04.121428] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:04.125752] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:08.126461] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:08.130777] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:12.131522] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:12.135960] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:16.136755] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:16.141071] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:20.141927] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:20.146272] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:24.147055] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:24.151407] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:28.152238] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:28.156586] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:32.157213] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:32.161542] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:36.162503] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:36.166872] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:40.167378] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:40.171725] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:44.172535] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:44.176884] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:48.177747] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:48.182084] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:52.182935] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:52.187334] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:34:56.187978] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:34:56.192483] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:00.193075] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:00.197618] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:04.198238] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:04.202567] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:08.203335] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:08.207657] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:12.208242] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:12.212615] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:16.213283] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:16.217824] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:20.218373] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:20.222696] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:24.223346] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:24.227742] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:28.228354] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:28.232836] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:32.233512] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:32.237982] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:36.238930] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:36.243299] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:40.243986] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:40.248339] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:44.249120] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:44.253443] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:48.254235] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:48.258575] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:52.259390] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:52.263705] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:35:56.264447] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:35:56.268897] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:00.269799] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:00.274177] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:04.274805] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:04.279309] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:08.280025] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:08.284436] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:12.285228] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:12.289661] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:16.290148] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:16.294497] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:20.295246] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:20.299590] E [socket.c:2169:socket_connect_finish] 0-test0-client-0: connection to 10.70.35.198:49187 failed (Connection refused) >[2014-08-18 10:36:24.999089] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-18 10:36:25.021930] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-18 10:36:25.118912] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-18 10:36:25.118947] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-18 10:36:25.119155] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-18 10:36:25.121076] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-18 10:36:25.122972] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-18 10:36:25.123077] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-18 10:36:25.123159] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-18 10:36:25.123372] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-18 10:36:25.127765] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-18 10:36:25.132021] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-18 10:36:25.136292] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-18 10:36:25.140573] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-18 10:36:25.144848] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume test0-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick0 > 6: option transport-type socket > 7: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 8: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 9: option send-gids true > 10: end-volume > 11: > 12: volume test0-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick0 > 17: option transport-type socket > 18: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 19: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 20: option send-gids true > 21: end-volume > 22: > 23: volume test0-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick0 > 28: option transport-type socket > 29: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 30: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 31: option send-gids true > 32: end-volume > 33: > 34: volume test0-dht > 35: type cluster/distribute > 36: subvolumes test0-client-0 test0-client-1 test0-client-2 > 37: end-volume > 38: > 39: volume test0-write-behind > 40: type performance/write-behind > 41: subvolumes test0-dht > 42: end-volume > 43: > 44: volume test0 > 45: type debug/io-stats > 46: option latency-measurement off > 47: option count-fop-hits off > 48: subvolumes test0-write-behind > 49: end-volume > 50: > 51: volume test1-client-0 > 52: type protocol/client > 53: option ping-timeout 42 > 54: option remote-host 10.70.35.198 > 55: option remote-subvolume /brick/1 > 56: option transport-type socket > 57: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 > 58: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb > 59: option send-gids true > 60: end-volume > 61: > 62: volume test1-client-1 > 63: type protocol/client > 64: option ping-timeout 42 > 65: option remote-host 10.70.35.198 > 66: option remote-subvolume /brick/2 > 67: option transport-type socket > 68: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 > 69: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb > 70: option send-gids true > 71: end-volume > 72: > 73: volume test1-client-2 > 74: type protocol/client > 75: option ping-timeout 42 > 76: option remote-host 10.70.35.198 > 77: option remote-subvolume /brick/3 > 78: option transport-type socket > 79: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 > 80: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb > 81: option send-gids true > 82: end-volume > 83: > 84: volume test1-dht > 85: type cluster/distribute > 86: subvolumes test1-client-0 test1-client-1 test1-client-2 > 87: end-volume > 88: > 89: volume test1-write-behind > 90: type performance/write-behind > 91: subvolumes test1-dht > 92: end-volume > 93: > 94: volume test1 > 95: type debug/io-stats > 96: option latency-measurement off > 97: option count-fop-hits off > 98: subvolumes test1-write-behind > 99: end-volume >100: >101: volume nfs-server >102: type nfs/server >103: option rpc-auth.auth-glusterfs on >104: option rpc-auth.auth-unix on >105: option rpc-auth.auth-null on >106: option rpc-auth.ports.insecure on >107: option rpc-auth-allow-insecure on >108: option transport-type socket >109: option transport.socket.listen-port 2049 >110: option nfs.dynamic-volumes on >111: option nfs.nlm on >112: option nfs.drc off >113: option rpc-auth.addr.test0.allow * >114: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >115: option rpc-auth.addr.test1.allow * >116: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >117: subvolumes test0 test1 >118: end-volume >119: >+------------------------------------------------------------------------------+ >[2014-08-18 10:36:25.150643] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-18 10:36:25.154965] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-18 10:36:25.155064] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-18 10:36:25.155149] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-18 10:36:25.168219] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:25.168555] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:25.168808] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-18 10:36:25.168830] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:25.169125] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-18 10:36:25.169147] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:25.169292] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:25.169364] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:25.169454] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-18 10:36:25.169495] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-18 10:36:25.169723] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-18 10:36:25.169745] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:25.169865] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-18 10:36:25.169885] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:25.170174] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-18 10:36:25.170235] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-18 10:36:26.049582] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-18 10:36:26.055168] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:26.055752] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-18 10:36:26.055789] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:26.056100] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-18 10:36:26.062471] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-18 10:36:26.069864] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-18 10:36:26.070434] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-18 10:36:26.070469] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-18 10:36:26.070992] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-18 10:36:33.882672] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-2: remote operation failed: File exists. Path: /f3 >[2014-08-18 10:37:06.932096] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-test0-client-2: remote operation failed: File exists. Path: /i1 >[2014-08-18 10:37:06.959533] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /f3: Rename: unlink on test0-client-2 failed [No such file or directory] >[2014-08-18 10:38:06.390964] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /i1: Rename: unlink on test0-client-2 failed [No such file or directory] >[2014-08-18 10:38:06.454534] W [MSGID: 109034] [dht-rename.c:402:dht_rename_unlink_cbk] 0-test0-dht: /i8: Rename: unlink on test0-client-0 failed [No such file or directory] >[2014-08-21 04:45:19.736574] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 04:45:19.738515] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 04:45:26.066129] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 04:45:26.087164] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 04:45:26.139000] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 04:45:26.139091] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 04:45:26.139317] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 04:45:26.141296] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 04:45:26.143269] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 04:45:26.145160] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 04:45:26.145277] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 04:45:26.145361] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 04:45:26.145605] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.149936] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.154247] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.158467] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.162721] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.166960] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.171201] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.175536] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 04:45:26.179831] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-dht > 35: type cluster/distribute > 36: subvolumes dist-client-0 dist-client-1 dist-client-2 > 37: end-volume > 38: > 39: volume dist-write-behind > 40: type performance/write-behind > 41: subvolumes dist-dht > 42: end-volume > 43: > 44: volume dist > 45: type debug/io-stats > 46: option latency-measurement off > 47: option count-fop-hits off > 48: subvolumes dist-write-behind > 49: end-volume > 50: > 51: volume test0-client-0 > 52: type protocol/client > 53: option ping-timeout 42 > 54: option remote-host 10.70.35.198 > 55: option remote-subvolume /brick0 > 56: option transport-type socket > 57: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 58: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 59: option send-gids true > 60: end-volume > 61: > 62: volume test0-client-1 > 63: type protocol/client > 64: option ping-timeout 42 > 65: option remote-host 10.70.35.240 > 66: option remote-subvolume /brick0 > 67: option transport-type socket > 68: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 69: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 70: option send-gids true > 71: end-volume > 72: > 73: volume test0-client-2 > 74: type protocol/client > 75: option ping-timeout 42 > 76: option remote-host 10.70.35.172 > 77: option remote-subvolume /brick0 > 78: option transport-type socket > 79: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 80: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 81: option send-gids true > 82: end-volume > 83: > 84: volume test0-dht > 85: type cluster/distribute > 86: subvolumes test0-client-0 test0-client-1 test0-client-2 > 87: end-volume > 88: > 89: volume test0-write-behind > 90: type performance/write-behind > 91: subvolumes test0-dht > 92: end-volume > 93: > 94: volume test0 > 95: type debug/io-stats > 96: option latency-measurement off > 97: option count-fop-hits off > 98: subvolumes test0-write-behind > 99: end-volume >100: >101: volume test1-client-0 >102: type protocol/client >103: option ping-timeout 42 >104: option remote-host 10.70.35.198 >105: option remote-subvolume /brick/1 >106: option transport-type socket >107: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >108: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >109: option send-gids true >110: end-volume >111: >112: volume test1-client-1 >113: type protocol/client >114: option ping-timeout 42 >115: option remote-host 10.70.35.198 >116: option remote-subvolume /brick/2 >117: option transport-type socket >118: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >119: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >120: option send-gids true >121: end-volume >122: >123: volume test1-client-2 >124: type protocol/client >125: option ping-timeout 42 >126: option remote-host 10.70.35.198 >127: option remote-subvolume /brick/3 >128: option transport-type socket >129: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >130: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >131: option send-gids true >132: end-volume >133: >134: volume test1-dht >135: type cluster/distribute >136: subvolumes test1-client-0 test1-client-1 test1-client-2 >137: end-volume >138: >139: volume test1-write-behind >140: type performance/write-behind >141: subvolumes test1-dht >142: end-volume >143: >144: volume test1 >145: type debug/io-stats >146: option latency-measurement off >147: option count-fop-hits off >148: subvolumes test1-write-behind >149: end-volume >150: >151: volume nfs-server >152: type nfs/server >153: option rpc-auth.auth-glusterfs on >154: option rpc-auth.auth-unix on >155: option rpc-auth.auth-null on >156: option rpc-auth.ports.insecure on >157: option rpc-auth-allow-insecure on >158: option transport-type socket >159: option transport.socket.listen-port 2049 >160: option nfs.dynamic-volumes on >161: option nfs.nlm on >162: option nfs.drc off >163: option rpc-auth.addr.dist.allow * >164: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >165: option rpc-auth.addr.test0.allow * >166: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >167: option rpc-auth.addr.test1.allow * >168: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >169: subvolumes dist test0 test1 >170: end-volume >171: >+------------------------------------------------------------------------------+ >[2014-08-21 04:45:26.186191] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 04:45:26.190544] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 04:45:26.190621] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 04:45:26.199264] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 04:45:26.199342] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-08-21 04:45:26.199396] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 04:45:26.199439] E [client-handshake.c:1498:client_query_portmap_cbk] 0-dist-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. >[2014-08-21 04:45:26.199481] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 04:45:26.199521] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 04:45:26.203994] I [client.c:2215:client_rpc_notify] 0-dist-client-1: disconnected from dist-client-1. Client process will keep trying to connect to glusterd until brick's port is available >[2014-08-21 04:45:26.208315] I [client.c:2215:client_rpc_notify] 0-dist-client-2: disconnected from dist-client-2. Client process will keep trying to connect to glusterd until brick's port is available >[2014-08-21 04:45:26.216817] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.217364] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 04:45:26.217387] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.217673] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.217906] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.221821] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 04:45:26.221865] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 04:45:26.221879] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.222187] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 04:45:26.222210] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.222791] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.222877] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 04:45:26.222904] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 04:45:26.222926] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.223014] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.223122] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:26.223253] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 04:45:26.223277] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.223416] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 04:45:26.223436] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.223920] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 04:45:26.223950] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 04:45:26.223978] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 04:45:26.223993] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.224126] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 04:45:26.224147] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:26.224358] I [dht-layout.c:663:dht_layout_normalize] 0-dist-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 >[2014-08-21 04:45:26.224402] W [MSGID: 109005] [dht-selfheal.c:1132:dht_selfheal_directory] 0-dist-dht: Directory selfheal failed: 2 subvolumes down.Not fixing. path = /, gfid = 00000000-0000-0000-0000-000000000001 >[2014-08-21 04:45:26.224775] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 04:45:26.224805] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 04:45:30.073680] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 04:45:30.082358] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 04:45:30.086780] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:30.087374] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 04:45:30.087400] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:30.087545] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 04:45:30.087764] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 04:45:30.088111] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 04:45:30.088133] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 04:45:30.088494] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 06:56:45.850495] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 06:56:45.856496] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 06:56:46.929717] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 06:56:46.952507] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 06:56:47.020854] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 06:56:47.020936] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 06:56:47.021173] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 06:56:47.023714] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 06:56:47.025663] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 06:56:47.027537] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 06:56:47.027645] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 06:56:47.027727] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 06:56:47.027990] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.032529] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.036874] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.041306] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.045591] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.049953] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.054319] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.058779] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.063097] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 06:56:47.067498] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-dht > 46: type cluster/distribute > 47: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 > 48: end-volume > 49: > 50: volume dist-write-behind > 51: type performance/write-behind > 52: subvolumes dist-dht > 53: end-volume > 54: > 55: volume dist > 56: type debug/io-stats > 57: option latency-measurement off > 58: option count-fop-hits off > 59: subvolumes dist-write-behind > 60: end-volume > 61: > 62: volume test0-client-0 > 63: type protocol/client > 64: option ping-timeout 42 > 65: option remote-host 10.70.35.198 > 66: option remote-subvolume /brick0 > 67: option transport-type socket > 68: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 69: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 70: option send-gids true > 71: end-volume > 72: > 73: volume test0-client-1 > 74: type protocol/client > 75: option ping-timeout 42 > 76: option remote-host 10.70.35.240 > 77: option remote-subvolume /brick0 > 78: option transport-type socket > 79: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 80: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 81: option send-gids true > 82: end-volume > 83: > 84: volume test0-client-2 > 85: type protocol/client > 86: option ping-timeout 42 > 87: option remote-host 10.70.35.172 > 88: option remote-subvolume /brick0 > 89: option transport-type socket > 90: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 91: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 92: option send-gids true > 93: end-volume > 94: > 95: volume test0-dht > 96: type cluster/distribute > 97: subvolumes test0-client-0 test0-client-1 test0-client-2 > 98: end-volume > 99: >100: volume test0-write-behind >101: type performance/write-behind >102: subvolumes test0-dht >103: end-volume >104: >105: volume test0 >106: type debug/io-stats >107: option latency-measurement off >108: option count-fop-hits off >109: subvolumes test0-write-behind >110: end-volume >111: >112: volume test1-client-0 >113: type protocol/client >114: option ping-timeout 42 >115: option remote-host 10.70.35.198 >116: option remote-subvolume /brick/1 >117: option transport-type socket >118: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >119: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >120: option send-gids true >121: end-volume >122: >123: volume test1-client-1 >124: type protocol/client >125: option ping-timeout 42 >126: option remote-host 10.70.35.198 >127: option remote-subvolume /brick/2 >128: option transport-type socket >129: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >130: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >131: option send-gids true >132: end-volume >133: >134: volume test1-client-2 >135: type protocol/client >136: option ping-timeout 42 >137: option remote-host 10.70.35.198 >138: option remote-subvolume /brick/3 >139: option transport-type socket >140: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >141: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >142: option send-gids true >143: end-volume >144: >145: volume test1-dht >146: type cluster/distribute >147: subvolumes test1-client-0 test1-client-1 test1-client-2 >148: end-volume >149: >150: volume test1-write-behind >151: type performance/write-behind >152: subvolumes test1-dht >153: end-volume >154: >155: volume test1 >156: type debug/io-stats >157: option latency-measurement off >158: option count-fop-hits off >159: subvolumes test1-write-behind >160: end-volume >161: >162: volume nfs-server >163: type nfs/server >164: option rpc-auth.auth-glusterfs on >165: option rpc-auth.auth-unix on >166: option rpc-auth.auth-null on >167: option rpc-auth.ports.insecure on >168: option rpc-auth-allow-insecure on >169: option transport-type socket >170: option transport.socket.listen-port 2049 >171: option nfs.dynamic-volumes on >172: option nfs.nlm on >173: option nfs.drc off >174: option rpc-auth.addr.dist.allow * >175: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >176: option rpc-auth.addr.test0.allow * >177: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >178: option rpc-auth.addr.test1.allow * >179: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >180: subvolumes dist test0 test1 >181: end-volume >182: >+------------------------------------------------------------------------------+ >[2014-08-21 06:56:47.074544] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 06:56:47.074625] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 06:56:47.074667] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 06:56:47.074703] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 06:56:47.093133] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 06:56:47.093227] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 06:56:47.102293] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.102406] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.102563] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.102899] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.103296] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 06:56:47.103319] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.103492] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 06:56:47.103512] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.103818] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 06:56:47.103839] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.104103] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.104343] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 06:56:47.104372] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 06:56:47.104393] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 06:56:47.104421] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 06:56:47.104472] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.104768] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.104893] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 06:56:47.105182] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 06:56:47.105205] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.105555] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 06:56:47.106615] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 06:56:47.106641] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.106881] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 06:56:47.110997] W [client-callback.c:31:client_cbk_fetchspec] 0-dist-client-1: this function should not be called >[2014-08-21 06:56:47.111059] W [client-callback.c:31:client_cbk_fetchspec] 0-test0-client-1: this function should not be called >[2014-08-21 06:56:47.114261] W [client-callback.c:31:client_cbk_fetchspec] 0-dist-client-2: this function should not be called >[2014-08-21 06:56:47.114293] W [client-callback.c:31:client_cbk_fetchspec] 0-test0-client-2: this function should not be called >[2014-08-21 06:56:47.149462] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 06:56:47.149547] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 06:56:47.160857] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.161108] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.161582] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 06:56:47.161639] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.162076] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 06:56:47.162117] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 06:56:47.162133] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.162444] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 06:56:47.170152] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 06:56:47.176253] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 06:56:47.181847] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.181953] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 06:56:47.182344] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 06:56:47.182370] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.182711] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 06:56:47.182750] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 06:56:47.182765] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 06:56:47.183072] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 07:11:21.123759] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 07:11:21.130022] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 07:11:22.200034] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 07:11:22.224135] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 07:11:22.336111] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 07:11:22.336181] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 07:11:22.336445] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:11:22.339352] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:11:22.341395] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:11:22.343574] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 07:11:22.343684] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 07:11:22.343772] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 07:11:22.344046] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.348740] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.353350] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.357804] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.362593] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.367217] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.371923] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.376330] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.381352] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.386485] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:11:22.391312] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-dht > 57: type cluster/distribute > 58: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 > 59: end-volume > 60: > 61: volume dist-write-behind > 62: type performance/write-behind > 63: subvolumes dist-dht > 64: end-volume > 65: > 66: volume dist > 67: type debug/io-stats > 68: option latency-measurement off > 69: option count-fop-hits off > 70: subvolumes dist-write-behind > 71: end-volume > 72: > 73: volume test0-client-0 > 74: type protocol/client > 75: option ping-timeout 42 > 76: option remote-host 10.70.35.198 > 77: option remote-subvolume /brick0 > 78: option transport-type socket > 79: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 80: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 81: option send-gids true > 82: end-volume > 83: > 84: volume test0-client-1 > 85: type protocol/client > 86: option ping-timeout 42 > 87: option remote-host 10.70.35.240 > 88: option remote-subvolume /brick0 > 89: option transport-type socket > 90: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 91: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 92: option send-gids true > 93: end-volume > 94: > 95: volume test0-client-2 > 96: type protocol/client > 97: option ping-timeout 42 > 98: option remote-host 10.70.35.172 > 99: option remote-subvolume /brick0 >100: option transport-type socket >101: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >102: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >103: option send-gids true >104: end-volume >105: >106: volume test0-dht >107: type cluster/distribute >108: subvolumes test0-client-0 test0-client-1 test0-client-2 >109: end-volume >110: >111: volume test0-write-behind >112: type performance/write-behind >113: subvolumes test0-dht >114: end-volume >115: >116: volume test0 >117: type debug/io-stats >118: option latency-measurement off >119: option count-fop-hits off >120: subvolumes test0-write-behind >121: end-volume >122: >123: volume test1-client-0 >124: type protocol/client >125: option ping-timeout 42 >126: option remote-host 10.70.35.198 >127: option remote-subvolume /brick/1 >128: option transport-type socket >129: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >130: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >131: option send-gids true >132: end-volume >133: >134: volume test1-client-1 >135: type protocol/client >136: option ping-timeout 42 >137: option remote-host 10.70.35.198 >138: option remote-subvolume /brick/2 >139: option transport-type socket >140: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >141: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >142: option send-gids true >143: end-volume >144: >145: volume test1-client-2 >146: type protocol/client >147: option ping-timeout 42 >148: option remote-host 10.70.35.198 >149: option remote-subvolume /brick/3 >150: option transport-type socket >151: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >152: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >153: option send-gids true >154: end-volume >155: >156: volume test1-dht >157: type cluster/distribute >158: subvolumes test1-client-0 test1-client-1 test1-client-2 >159: end-volume >160: >161: volume test1-write-behind >162: type performance/write-behind >163: subvolumes test1-dht >164: end-volume >165: >166: volume test1 >167: type debug/io-stats >168: option latency-measurement off >169: option count-fop-hits off >170: subvolumes test1-write-behind >171: end-volume >172: >173: volume nfs-server >174: type nfs/server >175: option rpc-auth.auth-glusterfs on >176: option rpc-auth.auth-unix on >177: option rpc-auth.auth-null on >178: option rpc-auth.ports.insecure on >179: option rpc-auth-allow-insecure on >180: option transport-type socket >181: option transport.socket.listen-port 2049 >182: option nfs.dynamic-volumes on >183: option nfs.nlm on >184: option nfs.drc off >185: option rpc-auth.addr.dist.allow * >186: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >187: option rpc-auth.addr.test0.allow * >188: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >189: option rpc-auth.addr.test1.allow * >190: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >191: subvolumes dist test0 test1 >192: end-volume >193: >+------------------------------------------------------------------------------+ >[2014-08-21 07:11:22.400459] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 07:11:22.400561] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 07:11:22.400653] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 07:11:22.400739] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 07:11:22.400828] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 07:11:22.400912] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 07:11:22.401018] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 07:11:22.435326] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.435473] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.435618] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.435758] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.436104] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.436430] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 07:11:22.436452] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.436609] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 07:11:22.436631] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.436749] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 07:11:22.436774] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.436893] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 07:11:22.436913] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.437250] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 07:11:22.437273] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.437576] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.437658] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:22.437752] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 07:11:22.437782] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 07:11:22.437801] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 07:11:22.437834] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 07:11:22.437853] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 07:11:22.438010] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 07:11:22.438031] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.438179] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 07:11:22.438201] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:22.438539] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 07:11:22.438570] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 07:11:23.281088] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 07:11:23.286816] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 07:11:23.291980] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:23.292250] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:23.292544] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 07:11:23.292564] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:23.292709] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 07:11:23.292737] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:23.293073] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 07:11:23.293142] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 07:11:23.300550] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 07:11:23.308471] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 07:11:23.313899] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:23.314015] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:11:23.314427] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 07:11:23.314458] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:23.314777] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 07:11:23.314803] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:11:23.315462] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 07:11:23.315495] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 07:13:28.666633] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 07:13:28.672548] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 07:13:29.759723] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 07:13:29.781938] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 07:13:29.896029] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 07:13:29.896069] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 07:13:29.896417] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:13:29.899694] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:13:29.901761] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:13:29.903825] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 07:13:29.903932] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 07:13:29.904045] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 07:13:29.904368] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.908983] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.913323] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.917693] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.921966] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.926260] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.930537] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.934857] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.939303] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.943824] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.948291] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:13:29.952780] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-dht > 68: type cluster/distribute > 69: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 > 70: end-volume > 71: > 72: volume dist-write-behind > 73: type performance/write-behind > 74: subvolumes dist-dht > 75: end-volume > 76: > 77: volume dist > 78: type debug/io-stats > 79: option latency-measurement off > 80: option count-fop-hits off > 81: subvolumes dist-write-behind > 82: end-volume > 83: > 84: volume test0-client-0 > 85: type protocol/client > 86: option ping-timeout 42 > 87: option remote-host 10.70.35.198 > 88: option remote-subvolume /brick0 > 89: option transport-type socket > 90: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 > 91: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd > 92: option send-gids true > 93: end-volume > 94: > 95: volume test0-client-1 > 96: type protocol/client > 97: option ping-timeout 42 > 98: option remote-host 10.70.35.240 > 99: option remote-subvolume /brick0 >100: option transport-type socket >101: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >102: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >103: option send-gids true >104: end-volume >105: >106: volume test0-client-2 >107: type protocol/client >108: option ping-timeout 42 >109: option remote-host 10.70.35.172 >110: option remote-subvolume /brick0 >111: option transport-type socket >112: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >113: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >114: option send-gids true >115: end-volume >116: >117: volume test0-dht >118: type cluster/distribute >119: subvolumes test0-client-0 test0-client-1 test0-client-2 >120: end-volume >121: >122: volume test0-write-behind >123: type performance/write-behind >124: subvolumes test0-dht >125: end-volume >126: >127: volume test0 >128: type debug/io-stats >129: option latency-measurement off >130: option count-fop-hits off >131: subvolumes test0-write-behind >132: end-volume >133: >134: volume test1-client-0 >135: type protocol/client >136: option ping-timeout 42 >137: option remote-host 10.70.35.198 >138: option remote-subvolume /brick/1 >139: option transport-type socket >140: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >141: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >142: option send-gids true >143: end-volume >144: >145: volume test1-client-1 >146: type protocol/client >147: option ping-timeout 42 >148: option remote-host 10.70.35.198 >149: option remote-subvolume /brick/2 >150: option transport-type socket >151: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >152: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >153: option send-gids true >154: end-volume >155: >156: volume test1-client-2 >157: type protocol/client >158: option ping-timeout 42 >159: option remote-host 10.70.35.198 >160: option remote-subvolume /brick/3 >161: option transport-type socket >162: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >163: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >164: option send-gids true >165: end-volume >166: >167: volume test1-dht >168: type cluster/distribute >169: subvolumes test1-client-0 test1-client-1 test1-client-2 >170: end-volume >171: >172: volume test1-write-behind >173: type performance/write-behind >174: subvolumes test1-dht >175: end-volume >176: >177: volume test1 >178: type debug/io-stats >179: option latency-measurement off >180: option count-fop-hits off >181: subvolumes test1-write-behind >182: end-volume >183: >184: volume nfs-server >185: type nfs/server >186: option rpc-auth.auth-glusterfs on >187: option rpc-auth.auth-unix on >188: option rpc-auth.auth-null on >189: option rpc-auth.ports.insecure on >190: option rpc-auth-allow-insecure on >191: option transport-type socket >192: option transport.socket.listen-port 2049 >193: option nfs.dynamic-volumes on >194: option nfs.nlm on >195: option nfs.drc off >196: option rpc-auth.addr.dist.allow * >197: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >198: option rpc-auth.addr.test0.allow * >199: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >200: option rpc-auth.addr.test1.allow * >201: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >202: subvolumes dist test0 test1 >203: end-volume >204: >+------------------------------------------------------------------------------+ >[2014-08-21 07:13:29.959891] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 07:13:29.959941] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 07:13:29.968795] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 07:13:29.968870] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 07:13:29.968912] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 07:13:29.982212] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 07:13:29.982266] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 07:13:29.982305] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 07:13:29.995671] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.995787] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.996095] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 07:13:29.996115] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.996267] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.996821] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 07:13:29.996878] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 07:13:29.996977] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.997292] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.997393] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.997660] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.997770] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 07:13:29.997791] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.997917] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.998019] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 07:13:29.998051] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 07:13:29.998064] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.998158] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 07:13:29.998178] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.998496] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 07:13:29.998518] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.998812] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 07:13:29.998846] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:29.998942] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 07:13:29.998969] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 07:13:29.999338] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 07:13:29.999363] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:29.999779] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 07:13:29.999812] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 07:13:30.001760] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 07:13:30.001798] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:30.002097] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 07:13:30.840426] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 07:13:30.845264] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 07:13:30.850348] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:30.850570] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:30.850866] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 07:13:30.851004] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:30.851136] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 07:13:30.851157] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:30.851291] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 07:13:30.851477] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 07:13:30.863003] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 07:13:30.869008] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 07:13:30.877461] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:30.877812] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:13:30.878441] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 07:13:30.878589] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:30.878909] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 07:13:30.878937] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:13:30.879264] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 07:13:30.879590] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 07:15:23.460723] W [rpcsvc.c:261:rpcsvc_program_actor] 0-rpc-service: RPC program version not available (req 100003 4) >[2014-08-21 07:15:23.460758] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >[2014-08-21 07:15:23.493916] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-0; inode layout - 3435973836 - 4294967295; disk layout - 3579139410 - 4294967295 >[2014-08-21 07:15:23.493960] I [MSGID: 109018] [dht-common.c:696:dht_revalidate_cbk] 0-dist-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001 >[2014-08-21 07:15:23.494032] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-1; inode layout - 0 - 858993458; disk layout - 2863311528 - 3579139409 >[2014-08-21 07:15:23.494074] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-4; inode layout - 2576980377 - 3435973835; disk layout - 0 - 715827881 >[2014-08-21 07:15:23.494110] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-2; inode layout - 1717986918 - 2576980376; disk layout - 2147483646 - 2863311527 >[2014-08-21 07:15:23.494144] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-3; inode layout - 858993459 - 1717986917; disk layout - 715827882 - 1431655763 >[2014-08-21 07:15:23.494198] I [dht-layout.c:782:dht_layout_dir_mismatch] 0-dist-dht: subvol: dist-client-5; inode layout - 0 - 0; disk layout - 1431655764 - 2147483645 >The message "I [MSGID: 109018] [dht-common.c:696:dht_revalidate_cbk] 0-dist-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 5 times between [2014-08-21 07:15:23.493960] and [2014-08-21 07:15:23.494215] >[2014-08-21 07:16:25.827782] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 07:16:25.834330] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 07:16:27.025493] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 07:16:27.050327] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 07:16:27.097261] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 07:16:27.097297] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 07:16:27.097524] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:16:27.102276] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:16:27.104239] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:16:27.106114] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 07:16:27.106250] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 07:16:27.106338] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 07:16:27.106692] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.111086] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.115357] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.119644] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.123928] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.128705] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.133085] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.137525] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.141954] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.146477] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.150845] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.155208] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.159474] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.163782] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.168073] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:16:27.172375] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-dht >112: type cluster/distribute >113: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 >114: end-volume >115: >116: volume dist-write-behind >117: type performance/write-behind >118: subvolumes dist-dht >119: end-volume >120: >121: volume dist >122: type debug/io-stats >123: option latency-measurement off >124: option count-fop-hits off >125: subvolumes dist-write-behind >126: end-volume >127: >128: volume test0-client-0 >129: type protocol/client >130: option ping-timeout 42 >131: option remote-host 10.70.35.198 >132: option remote-subvolume /brick0 >133: option transport-type socket >134: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >135: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >136: option send-gids true >137: end-volume >138: >139: volume test0-client-1 >140: type protocol/client >141: option ping-timeout 42 >142: option remote-host 10.70.35.240 >143: option remote-subvolume /brick0 >144: option transport-type socket >145: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >146: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >147: option send-gids true >148: end-volume >149: >150: volume test0-client-2 >151: type protocol/client >152: option ping-timeout 42 >153: option remote-host 10.70.35.172 >154: option remote-subvolume /brick0 >155: option transport-type socket >156: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >157: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >158: option send-gids true >159: end-volume >160: >161: volume test0-dht >162: type cluster/distribute >163: subvolumes test0-client-0 test0-client-1 test0-client-2 >164: end-volume >165: >166: volume test0-write-behind >167: type performance/write-behind >168: subvolumes test0-dht >169: end-volume >170: >171: volume test0 >172: type debug/io-stats >173: option latency-measurement off >174: option count-fop-hits off >175: subvolumes test0-write-behind >176: end-volume >177: >178: volume test1-client-0 >179: type protocol/client >180: option ping-timeout 42 >181: option remote-host 10.70.35.198 >182: option remote-subvolume /brick/1 >183: option transport-type socket >184: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >185: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >186: option send-gids true >187: end-volume >188: >189: volume test1-client-1 >190: type protocol/client >191: option ping-timeout 42 >192: option remote-host 10.70.35.198 >193: option remote-subvolume /brick/2 >194: option transport-type socket >195: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >196: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >197: option send-gids true >198: end-volume >199: >200: volume test1-client-2 >201: type protocol/client >202: option ping-timeout 42 >203: option remote-host 10.70.35.198 >204: option remote-subvolume /brick/3 >205: option transport-type socket >206: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >207: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >208: option send-gids true >209: end-volume >210: >211: volume test1-dht >212: type cluster/distribute >213: subvolumes test1-client-0 test1-client-1 test1-client-2 >214: end-volume >215: >216: volume test1-write-behind >217: type performance/write-behind >218: subvolumes test1-dht >219: end-volume >220: >221: volume test1 >222: type debug/io-stats >223: option latency-measurement off >224: option count-fop-hits off >225: subvolumes test1-write-behind >226: end-volume >227: >228: volume nfs-server >229: type nfs/server >230: option rpc-auth.auth-glusterfs on >231: option rpc-auth.auth-unix on >232: option rpc-auth.auth-null on >233: option rpc-auth.ports.insecure on >234: option rpc-auth-allow-insecure on >235: option transport-type socket >236: option transport.socket.listen-port 2049 >237: option nfs.dynamic-volumes on >238: option nfs.nlm on >239: option nfs.drc off >240: option rpc-auth.addr.dist.allow * >241: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >242: option rpc-auth.addr.test0.allow * >243: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >244: option rpc-auth.addr.test1.allow * >245: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >246: subvolumes dist test0 test1 >247: end-volume >248: >+------------------------------------------------------------------------------+ >[2014-08-21 07:16:27.179694] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 07:16:27.179744] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 07:16:27.179781] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 07:16:27.179815] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 07:16:27.179850] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 07:16:27.201716] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 07:16:27.201793] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 07:16:27.201835] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 07:16:27.201871] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 07:16:27.219285] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 07:16:27.219362] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 07:16:27.228420] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.228547] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.228645] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.228856] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.229338] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 07:16:27.229365] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.229519] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 07:16:27.229539] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.229653] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 07:16:27.229672] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.229958] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 07:16:27.229980] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.230098] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.232182] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.233169] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.233303] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.233543] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.233807] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 07:16:27.233839] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 07:16:27.233862] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 07:16:27.233886] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 07:16:27.233913] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 07:16:27.233926] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.234256] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 07:16:27.234280] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.234372] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 07:16:27.234390] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.234634] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.234745] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 07:16:27.234773] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 07:16:27.234794] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 07:16:27.234816] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.234911] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 07:16:27.234931] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.235102] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 07:16:27.235121] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.235287] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 07:16:27.235314] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.235626] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 07:16:27.235655] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 07:16:27.235680] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 07:16:27.236003] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 07:16:27.236026] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.236263] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 07:16:27.440545] W [client-callback.c:31:client_cbk_fetchspec] 0-dist-client-2: this function should not be called >[2014-08-21 07:16:27.440594] W [client-callback.c:31:client_cbk_fetchspec] 0-test0-client-2: this function should not be called >[2014-08-21 07:16:27.481657] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 07:16:27.481730] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 07:16:27.492040] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.492400] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:27.492561] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 07:16:27.492592] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.492929] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 07:16:27.492957] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:27.493118] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 07:16:27.493386] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 07:16:28.131734] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 07:16:28.131829] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 07:16:28.131864] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 07:16:28.143880] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:28.143973] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:28.144102] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:16:28.144615] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 07:16:28.144665] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:28.144763] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 07:16:28.144781] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:28.145218] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 07:16:28.145253] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 07:16:28.149200] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 07:16:28.149232] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:16:28.149703] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 07:16:32.043854] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-dist-client-6: remote operation failed: Stale file handle >[2014-08-21 07:16:32.046972] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-dist-client-7: remote operation failed: No such file or directory. Path: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> (05254230-2660-4c84-8163-02f6a427e5f7) >[2014-08-21 07:16:32.047121] E [dht-helper.c:813:dht_migration_complete_check_task] 0-dist-dht: <gfid:05254230-2660-4c84-8163-02f6a427e5f7>: failed to lookup the file on dist-client-7 >[2014-08-21 07:16:32.047159] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 70c8483: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> => -1 (No such file or directory) >[2014-08-21 07:16:32.047179] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 70c8483, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 07:16:32.048711] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-dist-client-7: remote operation failed: Stale file handle >[2014-08-21 07:16:32.051146] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-dist-client-6: remote operation failed: No such file or directory. Path: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> (05254230-2660-4c84-8163-02f6a427e5f7) >[2014-08-21 07:16:32.051299] E [dht-helper.c:813:dht_migration_complete_check_task] 0-dist-dht: <gfid:05254230-2660-4c84-8163-02f6a427e5f7>: failed to lookup the file on dist-client-6 >[2014-08-21 07:16:32.051342] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 80c8483: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> => -1 (No such file or directory) >[2014-08-21 07:16:32.051361] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 80c8483, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 07:16:32.052568] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-dist-client-6: remote operation failed: Stale file handle >[2014-08-21 07:16:32.056404] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-dist-client-6: remote operation failed: No such file or directory. Path: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> (05254230-2660-4c84-8163-02f6a427e5f7) >[2014-08-21 07:16:32.056554] E [dht-helper.c:813:dht_migration_complete_check_task] 0-dist-dht: <gfid:05254230-2660-4c84-8163-02f6a427e5f7>: failed to lookup the file on dist-client-6 >[2014-08-21 07:16:32.056587] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 90c8483: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> => -1 (No such file or directory) >[2014-08-21 07:16:32.056605] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 90c8483, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 07:16:32.057643] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-dist-client-6: remote operation failed: Stale file handle >[2014-08-21 07:16:32.060214] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-dist-client-6: remote operation failed: No such file or directory. Path: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> (05254230-2660-4c84-8163-02f6a427e5f7) >[2014-08-21 07:16:32.060370] E [dht-helper.c:813:dht_migration_complete_check_task] 0-dist-dht: <gfid:05254230-2660-4c84-8163-02f6a427e5f7>: failed to lookup the file on dist-client-6 >[2014-08-21 07:16:32.060402] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: a0c8483: <gfid:05254230-2660-4c84-8163-02f6a427e5f7> => -1 (No such file or directory) >[2014-08-21 07:16:32.060420] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: a0c8483, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 07:42:13.551518] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 07:42:13.563722] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 07:42:14.680214] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 07:42:14.702563] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 07:42:14.809266] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 07:42:14.809340] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 07:42:14.809546] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:42:14.814978] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:42:14.816828] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 07:42:14.818598] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 07:42:14.818688] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 07:42:14.818757] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 07:42:14.819084] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.822879] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.827076] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.831327] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.835695] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.840043] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.844559] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.849349] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.853814] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.857948] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.862542] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.867002] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.871568] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.875480] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.879915] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.884210] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.888702] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.893337] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 07:42:14.897382] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-dht >145: type cluster/distribute >146: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 >147: end-volume >148: >149: volume dist-write-behind >150: type performance/write-behind >151: subvolumes dist-dht >152: end-volume >153: >154: volume dist >155: type debug/io-stats >156: option latency-measurement off >157: option count-fop-hits off >158: subvolumes dist-write-behind >159: end-volume >160: >161: volume test0-client-0 >162: type protocol/client >163: option ping-timeout 42 >164: option remote-host 10.70.35.198 >165: option remote-subvolume /brick0 >166: option transport-type socket >167: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >168: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >169: option send-gids true >170: end-volume >171: >172: volume test0-client-1 >173: type protocol/client >174: option ping-timeout 42 >175: option remote-host 10.70.35.240 >176: option remote-subvolume /brick0 >177: option transport-type socket >178: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >179: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >180: option send-gids true >181: end-volume >182: >183: volume test0-client-2 >184: type protocol/client >185: option ping-timeout 42 >186: option remote-host 10.70.35.172 >187: option remote-subvolume /brick0 >188: option transport-type socket >189: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >190: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >191: option send-gids true >192: end-volume >193: >194: volume test0-dht >195: type cluster/distribute >196: subvolumes test0-client-0 test0-client-1 test0-client-2 >197: end-volume >198: >199: volume test0-write-behind >200: type performance/write-behind >201: subvolumes test0-dht >202: end-volume >203: >204: volume test0 >205: type debug/io-stats >206: option latency-measurement off >207: option count-fop-hits off >208: subvolumes test0-write-behind >209: end-volume >210: >211: volume test1-client-0 >212: type protocol/client >213: option ping-timeout 42 >214: option remote-host 10.70.35.198 >215: option remote-subvolume /brick/1 >216: option transport-type socket >217: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >218: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >219: option send-gids true >220: end-volume >221: >222: volume test1-client-1 >223: type protocol/client >224: option ping-timeout 42 >225: option remote-host 10.70.35.198 >226: option remote-subvolume /brick/2 >227: option transport-type socket >228: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >229: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >230: option send-gids true >231: end-volume >232: >233: volume test1-client-2 >234: type protocol/client >235: option ping-timeout 42 >236: option remote-host 10.70.35.198 >237: option remote-subvolume /brick/3 >238: option transport-type socket >239: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >240: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >241: option send-gids true >242: end-volume >243: >244: volume test1-dht >245: type cluster/distribute >246: subvolumes test1-client-0 test1-client-1 test1-client-2 >247: end-volume >248: >249: volume test1-write-behind >250: type performance/write-behind >251: subvolumes test1-dht >252: end-volume >253: >254: volume test1 >255: type debug/io-stats >256: option latency-measurement off >257: option count-fop-hits off >258: subvolumes test1-write-behind >259: end-volume >260: >261: volume nfs-server >262: type nfs/server >263: option rpc-auth.auth-glusterfs on >264: option rpc-auth.auth-unix on >265: option rpc-auth.auth-null on >266: option rpc-auth.ports.insecure on >267: option rpc-auth-allow-insecure on >268: option transport-type socket >269: option transport.socket.listen-port 2049 >270: option nfs.dynamic-volumes on >271: option nfs.nlm on >272: option nfs.drc off >273: option rpc-auth.addr.dist.allow * >274: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >275: option rpc-auth.addr.test0.allow * >276: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >277: option rpc-auth.addr.test1.allow * >278: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >279: subvolumes dist test0 test1 >280: end-volume >281: >+------------------------------------------------------------------------------+ >[2014-08-21 07:42:14.904464] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 07:42:14.904516] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 07:42:14.904553] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 07:42:14.904588] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 07:42:14.904623] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 07:42:14.904657] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 07:42:14.904720] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 07:42:14.904760] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 07:42:14.946486] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 07:42:14.946542] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 07:42:14.946581] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 07:42:14.946617] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 07:42:14.965117] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.965266] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.965372] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.965530] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.965836] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.965941] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.966033] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.966690] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 07:42:14.966716] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.966899] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 07:42:14.966929] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.967264] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 07:42:14.967296] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.967483] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 07:42:14.967505] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.967816] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 07:42:14.967838] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.967959] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 07:42:14.967979] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.968097] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 07:42:14.968233] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.968605] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.968738] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.968977] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.969085] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.969265] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 07:42:14.969307] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 07:42:14.969347] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 07:42:14.969386] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 07:42:14.969433] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 07:42:14.969455] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 07:42:14.969475] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 07:42:14.969504] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 07:42:14.969519] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.969651] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 07:42:14.969672] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.969954] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:14.970141] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 07:42:14.970173] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 07:42:14.970235] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 07:42:14.970251] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.970422] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 07:42:14.970453] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.970776] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 07:42:14.970833] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 07:42:14.970860] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:14.971259] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 07:42:14.971306] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 07:42:15.896374] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 07:42:15.902714] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 07:42:15.910114] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 07:42:15.917258] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.917512] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.917586] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.917817] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 07:42:15.917840] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.918523] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 07:42:15.918548] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.919021] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 07:42:15.919158] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 07:42:15.921907] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 07:42:15.921940] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.922291] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 07:42:15.943226] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 07:42:15.947775] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 07:42:15.947837] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 07:42:15.956776] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 07:42:15.961567] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.961665] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.961742] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.961820] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 07:42:15.962092] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 07:42:15.962115] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.962399] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 07:42:15.962421] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.962536] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 07:42:15.962571] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 07:42:15.962587] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.963132] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 07:42:15.963223] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 07:42:15.966288] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 07:42:15.966320] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 07:42:15.966638] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:00:01.470028] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:00:01.478898] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:00:02.591780] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:00:02.617384] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:00:02.703086] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:00:02.703177] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:00:02.703541] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:00:02.712153] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:00:02.714159] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:00:02.716082] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:00:02.716189] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:00:02.716283] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:00:02.716737] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.721129] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.725401] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.729894] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.734216] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.738623] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.743487] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.747772] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.752349] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.757135] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.762125] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.767996] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.774307] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.779704] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.784957] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.790131] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.794930] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.799689] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.804628] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.809195] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.813958] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:00:02.818729] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-dht >178: type cluster/distribute >179: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 >180: end-volume >181: >182: volume dist-write-behind >183: type performance/write-behind >184: subvolumes dist-dht >185: end-volume >186: >187: volume dist >188: type debug/io-stats >189: option latency-measurement off >190: option count-fop-hits off >191: subvolumes dist-write-behind >192: end-volume >193: >194: volume test0-client-0 >195: type protocol/client >196: option ping-timeout 42 >197: option remote-host 10.70.35.198 >198: option remote-subvolume /brick0 >199: option transport-type socket >200: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >201: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >202: option send-gids true >203: end-volume >204: >205: volume test0-client-1 >206: type protocol/client >207: option ping-timeout 42 >208: option remote-host 10.70.35.240 >209: option remote-subvolume /brick0 >210: option transport-type socket >211: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >212: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >213: option send-gids true >214: end-volume >215: >216: volume test0-client-2 >217: type protocol/client >218: option ping-timeout 42 >219: option remote-host 10.70.35.172 >220: option remote-subvolume /brick0 >221: option transport-type socket >222: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >223: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >224: option send-gids true >225: end-volume >226: >227: volume test0-dht >228: type cluster/distribute >229: subvolumes test0-client-0 test0-client-1 test0-client-2 >230: end-volume >231: >232: volume test0-write-behind >233: type performance/write-behind >234: subvolumes test0-dht >235: end-volume >236: >237: volume test0 >238: type debug/io-stats >239: option latency-measurement off >240: option count-fop-hits off >241: subvolumes test0-write-behind >242: end-volume >243: >244: volume test1-client-0 >245: type protocol/client >246: option ping-timeout 42 >247: option remote-host 10.70.35.198 >248: option remote-subvolume /brick/1 >249: option transport-type socket >250: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >251: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >252: option send-gids true >253: end-volume >254: >255: volume test1-client-1 >256: type protocol/client >257: option ping-timeout 42 >258: option remote-host 10.70.35.198 >259: option remote-subvolume /brick/2 >260: option transport-type socket >261: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >262: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >263: option send-gids true >264: end-volume >265: >266: volume test1-client-2 >267: type protocol/client >268: option ping-timeout 42 >269: option remote-host 10.70.35.198 >270: option remote-subvolume /brick/3 >271: option transport-type socket >272: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >273: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >274: option send-gids true >275: end-volume >276: >277: volume test1-dht >278: type cluster/distribute >279: subvolumes test1-client-0 test1-client-1 test1-client-2 >280: end-volume >281: >282: volume test1-write-behind >283: type performance/write-behind >284: subvolumes test1-dht >285: end-volume >286: >287: volume test1 >288: type debug/io-stats >289: option latency-measurement off >290: option count-fop-hits off >291: subvolumes test1-write-behind >292: end-volume >293: >294: volume nfs-server >295: type nfs/server >296: option rpc-auth.auth-glusterfs on >297: option rpc-auth.auth-unix on >298: option rpc-auth.auth-null on >299: option rpc-auth.ports.insecure on >300: option rpc-auth-allow-insecure on >301: option transport-type socket >302: option transport.socket.listen-port 2049 >303: option nfs.dynamic-volumes on >304: option nfs.nlm on >305: option nfs.drc off >306: option rpc-auth.addr.dist.allow * >307: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >308: option rpc-auth.addr.test0.allow * >309: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >310: option rpc-auth.addr.test1.allow * >311: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >312: subvolumes dist test0 test1 >313: end-volume >314: >+------------------------------------------------------------------------------+ >[2014-08-21 08:00:02.828189] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:00:02.828292] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:00:02.828337] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:00:02.828373] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:00:02.828408] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:00:02.828472] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:00:02.828512] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:00:02.828547] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:00:02.866810] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:00:02.871731] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:00:02.871829] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:00:02.871915] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:00:02.872017] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:00:02.892164] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.892747] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.892848] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.892942] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.893033] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.893104] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.893242] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:00:02.893277] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.893418] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:00:02.893465] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.893625] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:00:02.893651] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.893782] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:00:02.893802] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.893924] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.894022] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.894472] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:00:02.894506] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:00:02.894527] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:00:02.894547] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:00:02.894576] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:00:02.894590] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.894713] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:00:02.894733] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.894848] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.895081] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.895199] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.895559] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.895823] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:00:02.895844] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.895956] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:00:02.895976] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.896138] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:00:02.896166] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:00:02.896193] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:00:02.896208] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.896598] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:00:02.896631] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.896825] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:00:02.896848] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.897151] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:00:02.897173] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.897305] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:00:02.897338] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:02.897553] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:00:02.897592] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:00:02.897635] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:00:02.897675] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:00:02.897727] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:00:02.897926] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:00:02.897948] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:02.898475] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:00:03.780748] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:00:03.784764] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:00:03.784883] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:00:03.784940] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:00:03.800117] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.800257] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.800337] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.800404] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.800569] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:00:03.800593] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.800758] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:00:03.800827] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.801390] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:00:03.801412] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.801537] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:00:03.801832] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:00:03.802068] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:00:03.804677] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:00:03.804717] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.805115] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:00:03.805166] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:00:03.811019] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:00:03.815815] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:00:03.815895] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:00:03.815945] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:00:03.830928] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.831033] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.831109] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.831175] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.831261] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:00:03.831413] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:00:03.831440] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.831827] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:00:03.831849] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.831986] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:00:03.835896] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:00:03.835951] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.836179] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:00:03.836235] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:00:03.836337] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.836646] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:00:03.836688] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:00:03.836760] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:00:03.836778] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:00:03.837515] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:01:23.507518] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:01:23.516676] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:01:24.630697] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:01:24.651432] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:01:24.699035] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:01:24.699156] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:01:24.699424] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:01:24.708073] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:01:24.710069] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:01:24.712034] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:01:24.712141] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:01:24.712235] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:01:24.712776] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.717373] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.721642] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.725983] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.730376] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.734644] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.739004] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.743428] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.747752] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.752052] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.756395] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.760738] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.764980] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.769415] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.773853] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.778175] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.782530] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.786929] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.791288] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.796457] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.800842] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.805096] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.809476] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.813802] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:01:24.818124] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-dht >211: type cluster/distribute >212: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 >213: end-volume >214: >215: volume dist-write-behind >216: type performance/write-behind >217: subvolumes dist-dht >218: end-volume >219: >220: volume dist >221: type debug/io-stats >222: option latency-measurement off >223: option count-fop-hits off >224: subvolumes dist-write-behind >225: end-volume >226: >227: volume test0-client-0 >228: type protocol/client >229: option ping-timeout 42 >230: option remote-host 10.70.35.198 >231: option remote-subvolume /brick0 >232: option transport-type socket >233: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >234: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >235: option send-gids true >236: end-volume >237: >238: volume test0-client-1 >239: type protocol/client >240: option ping-timeout 42 >241: option remote-host 10.70.35.240 >242: option remote-subvolume /brick0 >243: option transport-type socket >244: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >245: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >246: option send-gids true >247: end-volume >248: >249: volume test0-client-2 >250: type protocol/client >251: option ping-timeout 42 >252: option remote-host 10.70.35.172 >253: option remote-subvolume /brick0 >254: option transport-type socket >255: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >256: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >257: option send-gids true >258: end-volume >259: >260: volume test0-dht >261: type cluster/distribute >262: subvolumes test0-client-0 test0-client-1 test0-client-2 >263: end-volume >264: >265: volume test0-write-behind >266: type performance/write-behind >267: subvolumes test0-dht >268: end-volume >269: >270: volume test0 >271: type debug/io-stats >272: option latency-measurement off >273: option count-fop-hits off >274: subvolumes test0-write-behind >275: end-volume >276: >277: volume test1-client-0 >278: type protocol/client >279: option ping-timeout 42 >280: option remote-host 10.70.35.198 >281: option remote-subvolume /brick/1 >282: option transport-type socket >283: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >284: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >285: option send-gids true >286: end-volume >287: >288: volume test1-client-1 >289: type protocol/client >290: option ping-timeout 42 >291: option remote-host 10.70.35.198 >292: option remote-subvolume /brick/2 >293: option transport-type socket >294: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >295: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >296: option send-gids true >297: end-volume >298: >299: volume test1-client-2 >300: type protocol/client >301: option ping-timeout 42 >302: option remote-host 10.70.35.198 >303: option remote-subvolume /brick/3 >304: option transport-type socket >305: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >306: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >307: option send-gids true >308: end-volume >309: >310: volume test1-dht >311: type cluster/distribute >312: subvolumes test1-client-0 test1-client-1 test1-client-2 >313: end-volume >314: >315: volume test1-write-behind >316: type performance/write-behind >317: subvolumes test1-dht >318: end-volume >319: >320: volume test1 >321: type debug/io-stats >322: option latency-measurement off >323: option count-fop-hits off >324: subvolumes test1-write-behind >325: end-volume >326: >327: volume nfs-server >328: type nfs/server >329: option rpc-auth.auth-glusterfs on >330: option rpc-auth.auth-unix on >331: option rpc-auth.auth-null on >332: option rpc-auth.ports.insecure on >333: option rpc-auth-allow-insecure on >334: option transport-type socket >335: option transport.socket.listen-port 2049 >336: option nfs.dynamic-volumes on >337: option nfs.nlm on >338: option nfs.drc off >339: option rpc-auth.addr.dist.allow * >340: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >341: option rpc-auth.addr.test0.allow * >342: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >343: option rpc-auth.addr.test1.allow * >344: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >345: subvolumes dist test0 test1 >346: end-volume >347: >+------------------------------------------------------------------------------+ >[2014-08-21 08:01:24.826782] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:01:24.826832] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:01:24.826868] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:01:24.826902] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:01:24.826935] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:01:24.826968] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:01:24.827002] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:01:24.827045] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:01:24.827108] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:01:24.867058] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:01:24.867125] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:01:24.867164] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:01:24.867198] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:01:24.867249] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:01:24.890666] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.890763] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.891003] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.891249] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.891371] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.891441] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.892171] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:01:24.892194] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.892368] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:01:24.892389] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.892797] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:01:24.892819] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.893130] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:01:24.893152] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.893290] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:01:24.893338] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.893436] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:01:24.893455] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.893562] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.893632] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.893869] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.893970] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.894062] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.894158] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:01:24.894187] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:01:24.894208] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:01:24.894245] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:01:24.894266] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:01:24.894286] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:01:24.894339] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:01:24.894357] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.894481] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:01:24.894501] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.894746] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.894845] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.894914] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:24.895026] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:01:24.895053] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:01:24.895084] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:01:24.895099] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.895235] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:01:24.895258] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.895387] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:01:24.895408] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.895644] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:01:24.895672] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:01:24.895699] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:01:24.895746] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.895886] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:01:24.895939] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.896071] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:01:24.896090] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:24.896529] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:01:24.896561] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:01:24.896581] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:01:24.896601] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:01:25.798233] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:01:25.804678] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:01:25.810299] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:01:25.810446] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:01:25.824248] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:01:25.824417] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.830500] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.830733] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.830925] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:01:25.830955] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.831284] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.831504] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:01:25.831536] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.831751] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.831946] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:01:25.831976] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.835262] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:01:25.835362] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.835580] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:01:25.835923] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:01:25.836236] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:01:25.836591] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:01:25.837780] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:01:25.837809] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.838149] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:01:25.883130] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:01:25.883212] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:01:25.893223] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:01:25.893289] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:01:25.902274] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:01:25.902329] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:01:25.911581] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.911951] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.912082] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.912192] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.912347] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.912565] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:01:25.912692] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:01:25.912721] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.912861] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:01:25.912882] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.913524] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:01:25.913548] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.914054] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:01:25.914086] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:01:25.914107] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:01:25.914136] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:01:25.914150] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.914355] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:01:25.914378] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.914917] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:01:25.914949] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:01:25.917763] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:01:25.917792] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:01:25.918162] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:04:05.363082] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:04:05.370432] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:04:06.499792] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:04:06.522000] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:04:06.574216] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:04:06.574259] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:04:06.574570] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:04:06.585574] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:04:06.587642] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:04:06.589546] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:04:06.589643] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:04:06.589712] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:04:06.590190] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.594630] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.598949] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.603089] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.607326] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.611759] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.615939] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.620203] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.624536] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.628734] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.633078] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.637392] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.641620] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.645919] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.650157] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.654332] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.658613] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.662963] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.667213] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.671615] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.675868] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.680062] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.684768] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.689315] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.693975] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.698523] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.703123] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:04:06.707668] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-dht >244: type cluster/distribute >245: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 >246: end-volume >247: >248: volume dist-write-behind >249: type performance/write-behind >250: subvolumes dist-dht >251: end-volume >252: >253: volume dist >254: type debug/io-stats >255: option latency-measurement off >256: option count-fop-hits off >257: subvolumes dist-write-behind >258: end-volume >259: >260: volume test0-client-0 >261: type protocol/client >262: option ping-timeout 42 >263: option remote-host 10.70.35.198 >264: option remote-subvolume /brick0 >265: option transport-type socket >266: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >267: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >268: option send-gids true >269: end-volume >270: >271: volume test0-client-1 >272: type protocol/client >273: option ping-timeout 42 >274: option remote-host 10.70.35.240 >275: option remote-subvolume /brick0 >276: option transport-type socket >277: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >278: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >279: option send-gids true >280: end-volume >281: >282: volume test0-client-2 >283: type protocol/client >284: option ping-timeout 42 >285: option remote-host 10.70.35.172 >286: option remote-subvolume /brick0 >287: option transport-type socket >288: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >289: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >290: option send-gids true >291: end-volume >292: >293: volume test0-dht >294: type cluster/distribute >295: subvolumes test0-client-0 test0-client-1 test0-client-2 >296: end-volume >297: >298: volume test0-write-behind >299: type performance/write-behind >300: subvolumes test0-dht >301: end-volume >302: >303: volume test0 >304: type debug/io-stats >305: option latency-measurement off >306: option count-fop-hits off >307: subvolumes test0-write-behind >308: end-volume >309: >310: volume test1-client-0 >311: type protocol/client >312: option ping-timeout 42 >313: option remote-host 10.70.35.198 >314: option remote-subvolume /brick/1 >315: option transport-type socket >316: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >317: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >318: option send-gids true >319: end-volume >320: >321: volume test1-client-1 >322: type protocol/client >323: option ping-timeout 42 >324: option remote-host 10.70.35.198 >325: option remote-subvolume /brick/2 >326: option transport-type socket >327: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >328: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >329: option send-gids true >330: end-volume >331: >332: volume test1-client-2 >333: type protocol/client >334: option ping-timeout 42 >335: option remote-host 10.70.35.198 >336: option remote-subvolume /brick/3 >337: option transport-type socket >338: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >339: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >340: option send-gids true >341: end-volume >342: >343: volume test1-dht >344: type cluster/distribute >345: subvolumes test1-client-0 test1-client-1 test1-client-2 >346: end-volume >347: >348: volume test1-write-behind >349: type performance/write-behind >350: subvolumes test1-dht >351: end-volume >352: >353: volume test1 >354: type debug/io-stats >355: option latency-measurement off >356: option count-fop-hits off >357: subvolumes test1-write-behind >358: end-volume >359: >360: volume nfs-server >361: type nfs/server >362: option rpc-auth.auth-glusterfs on >363: option rpc-auth.auth-unix on >364: option rpc-auth.auth-null on >365: option rpc-auth.ports.insecure on >366: option rpc-auth-allow-insecure on >367: option transport-type socket >368: option transport.socket.listen-port 2049 >369: option nfs.dynamic-volumes on >370: option nfs.nlm on >371: option nfs.drc off >372: option rpc-auth.addr.dist.allow * >373: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >374: option rpc-auth.addr.test0.allow * >375: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >376: option rpc-auth.addr.test1.allow * >377: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >378: subvolumes dist test0 test1 >379: end-volume >380: >+------------------------------------------------------------------------------+ >[2014-08-21 08:04:06.718972] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:04:06.719065] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:04:06.719142] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:04:06.719226] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:04:06.719300] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:04:06.719374] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:04:06.719483] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:04:06.719563] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:04:06.719638] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:04:06.719733] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:04:06.719848] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:04:06.719943] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:04:06.720020] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:04:06.720093] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:04:06.720166] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:04:06.785793] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.785901] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.785963] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786023] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786102] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786162] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786223] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786278] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786334] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786389] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786483] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786545] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.786768] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:04:06.786787] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.786910] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:04:06.786928] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787003] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:04:06.787028] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787151] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:04:06.787178] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787294] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:04:06.787313] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787390] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:04:06.787407] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787528] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:04:06.787546] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787624] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:04:06.787640] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787714] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:04:06.787731] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787807] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:04:06.787823] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.787899] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:04:06.787916] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.788135] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.788201] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.788258] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:06.788318] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:04:06.788341] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:04:06.788359] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:04:06.788376] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:04:06.788393] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:04:06.788410] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:04:06.788479] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:04:06.788517] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:04:06.788550] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:04:06.788586] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:04:06.788623] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:04:06.788671] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:04:06.788687] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.788789] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:04:06.788807] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.788971] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:04:06.789001] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:04:06.789014] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.789094] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:04:06.789111] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:06.789622] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:04:06.789649] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:04:06.789667] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:04:07.712257] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:04:07.716955] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:04:07.717082] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:04:07.717166] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:04:07.717251] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:04:07.734437] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:04:07.739469] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.739624] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.739722] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:04:07.739836] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.740008] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:04:07.740045] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.744475] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:04:07.744524] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:04:07.744588] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.744681] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:04:07.744722] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:04:07.744781] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:04:07.744822] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:04:07.744898] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:04:07.767180] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.767269] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.767394] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:04:07.767412] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.767495] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:04:07.767512] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.767651] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:04:07.767668] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.768121] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768263] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768430] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:04:07.768450] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.768585] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:04:07.768623] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:04:07.768642] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:04:07.768694] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768761] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768829] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768923] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.768989] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:04:07.769156] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:04:07.769174] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.769274] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:04:07.769290] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.769427] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:04:07.769539] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:04:07.769579] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.769698] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:04:07.769715] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.770005] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:04:07.770025] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.770188] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:04:07.770215] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:04:07.770241] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:04:07.770253] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.770367] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:04:07.770393] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:04:07.770551] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:04:07.770795] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:04:07.772171] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:04:07.772198] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.772573] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:04:07.773034] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:04:07.773054] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:04:07.773512] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:05:44.560630] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:05:44.568714] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:05:45.695677] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:05:45.721046] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:05:45.776971] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:05:45.777073] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:05:45.777275] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:05:45.788684] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:05:45.790825] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:05:45.792677] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:05:45.792775] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:05:45.792847] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:05:45.793360] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.797863] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.801809] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.806360] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.810216] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.814302] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.818253] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.822307] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.826241] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.830297] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.834289] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.838396] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.842922] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.847028] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.850999] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.854786] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.858695] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.862636] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.866664] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.870738] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.874818] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.878720] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.882885] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.886836] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.890739] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.894546] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.898500] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.902360] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.906337] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.910374] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:05:45.914214] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-dht >277: type cluster/distribute >278: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 >279: end-volume >280: >281: volume dist-write-behind >282: type performance/write-behind >283: subvolumes dist-dht >284: end-volume >285: >286: volume dist >287: type debug/io-stats >288: option latency-measurement off >289: option count-fop-hits off >290: subvolumes dist-write-behind >291: end-volume >292: >293: volume test0-client-0 >294: type protocol/client >295: option ping-timeout 42 >296: option remote-host 10.70.35.198 >297: option remote-subvolume /brick0 >298: option transport-type socket >299: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >300: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >301: option send-gids true >302: end-volume >303: >304: volume test0-client-1 >305: type protocol/client >306: option ping-timeout 42 >307: option remote-host 10.70.35.240 >308: option remote-subvolume /brick0 >309: option transport-type socket >310: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >311: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >312: option send-gids true >313: end-volume >314: >315: volume test0-client-2 >316: type protocol/client >317: option ping-timeout 42 >318: option remote-host 10.70.35.172 >319: option remote-subvolume /brick0 >320: option transport-type socket >321: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >322: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >323: option send-gids true >324: end-volume >325: >326: volume test0-dht >327: type cluster/distribute >328: subvolumes test0-client-0 test0-client-1 test0-client-2 >329: end-volume >330: >331: volume test0-write-behind >332: type performance/write-behind >333: subvolumes test0-dht >334: end-volume >335: >336: volume test0 >337: type debug/io-stats >338: option latency-measurement off >339: option count-fop-hits off >340: subvolumes test0-write-behind >341: end-volume >342: >343: volume test1-client-0 >344: type protocol/client >345: option ping-timeout 42 >346: option remote-host 10.70.35.198 >347: option remote-subvolume /brick/1 >348: option transport-type socket >349: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >350: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >351: option send-gids true >352: end-volume >353: >354: volume test1-client-1 >355: type protocol/client >356: option ping-timeout 42 >357: option remote-host 10.70.35.198 >358: option remote-subvolume /brick/2 >359: option transport-type socket >360: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >361: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >362: option send-gids true >363: end-volume >364: >365: volume test1-client-2 >366: type protocol/client >367: option ping-timeout 42 >368: option remote-host 10.70.35.198 >369: option remote-subvolume /brick/3 >370: option transport-type socket >371: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >372: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >373: option send-gids true >374: end-volume >375: >376: volume test1-dht >377: type cluster/distribute >378: subvolumes test1-client-0 test1-client-1 test1-client-2 >379: end-volume >380: >381: volume test1-write-behind >382: type performance/write-behind >383: subvolumes test1-dht >384: end-volume >385: >386: volume test1 >387: type debug/io-stats >388: option latency-measurement off >389: option count-fop-hits off >390: subvolumes test1-write-behind >391: end-volume >392: >393: volume nfs-server >394: type nfs/server >395: option rpc-auth.auth-glusterfs on >396: option rpc-auth.auth-unix on >397: option rpc-auth.auth-null on >398: option rpc-auth.ports.insecure on >399: option rpc-auth-allow-insecure on >400: option transport-type socket >401: option transport.socket.listen-port 2049 >402: option nfs.dynamic-volumes on >403: option nfs.nlm on >404: option nfs.drc off >405: option rpc-auth.addr.dist.allow * >406: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >407: option rpc-auth.addr.test0.allow * >408: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >409: option rpc-auth.addr.test1.allow * >410: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >411: subvolumes dist test0 test1 >412: end-volume >413: >+------------------------------------------------------------------------------+ >[2014-08-21 08:05:45.925286] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:05:45.925390] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:05:45.925485] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:05:45.925564] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:05:45.925642] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:05:45.925717] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:05:45.925798] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:05:45.925873] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:05:45.925946] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:05:45.926025] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:05:45.926153] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:05:45.926235] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 08:05:45.926314] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:05:45.926390] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:05:45.926493] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:05:45.926574] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:05:45.989534] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.989630] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.989723] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.989795] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.989855] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.989935] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990002] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990086] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990157] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990356] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990453] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990536] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990597] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990675] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990769] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.990989] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:05:45.991008] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991155] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:05:45.991176] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991290] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:05:45.991307] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991388] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:05:45.991416] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991504] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:05:45.991521] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991623] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:05:45.991640] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991718] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:05:45.991734] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991811] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:05:45.991827] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.991909] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:05:45.991927] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992184] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:05:45.992205] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992287] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:05:45.992304] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992389] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 08:05:45.992420] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992512] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:05:45.992532] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992618] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:05:45.992636] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.992859] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:05:45.993003] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.993355] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:45.993482] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:05:45.993509] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:05:45.993527] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:05:45.993545] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:05:45.993562] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:05:45.993579] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:05:45.993596] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:05:45.993613] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:05:45.993631] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:05:45.993648] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:05:45.993665] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:05:45.993682] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 08:05:45.993699] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:05:45.993716] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:05:45.993735] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:05:45.993869] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:05:45.993890] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:45.994519] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:05:46.920661] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:05:46.925568] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:05:46.925630] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:05:46.925675] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:05:46.925709] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:05:46.925742] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 08:05:46.925973] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:05:46.961902] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:05:46.961990] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:05:46.962037] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:05:46.962082] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:05:46.962368] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:05:46.962455] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:05:46.962510] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 08:05:46.962543] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:05:46.962830] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:46.962938] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.006395] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.006573] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:05:47.006595] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.006711] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:05:47.006731] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.006832] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.006908] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.007095] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.007774] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:05:47.007829] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:05:47.007888] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:05:47.007904] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.008130] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:05:47.008151] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.008329] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.008430] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.008502] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.008624] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:05:47.008643] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.008773] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:05:47.008802] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:05:47.008825] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.008927] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.008997] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.009066] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.009168] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.009240] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:05:47.009452] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:05:47.009474] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.009606] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:05:47.009626] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.009752] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:05:47.009860] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:05:47.009881] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.010076] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:05:47.010126] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:05:47.010167] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:05:47.010183] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.010368] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:05:47.010389] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.010916] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:05:47.010940] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.011096] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:05:47.011124] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.011264] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:05:47.011297] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.011569] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:05:47.011634] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:05:47.011659] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:05:47.011680] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:05:47.011940] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 08:05:47.011962] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.012230] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:05:47.012260] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:05:47.012596] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 08:05:47.013270] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 08:05:47.013477] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:05:47.013867] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 08:29:44.958356] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:29:44.969595] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:29:46.124963] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:29:46.147827] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:29:46.267591] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:29:46.267659] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:29:46.267920] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:29:46.284363] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:29:46.287312] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:29:46.290027] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:29:46.290154] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:29:46.290284] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:29:46.290965] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.296502] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.301944] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.307341] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.312247] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.317336] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.322473] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.327556] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.332596] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.337781] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.342820] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.347719] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.352327] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.356732] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.361610] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.367348] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.371798] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.376513] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.380833] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.385101] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.389641] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.394876] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.399760] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.404509] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.409285] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.414043] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.418892] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.423528] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.428074] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.432730] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.437281] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.441625] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.446215] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:29:46.450788] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-dht >310: type cluster/distribute >311: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 >312: end-volume >313: >314: volume dist-write-behind >315: type performance/write-behind >316: subvolumes dist-dht >317: end-volume >318: >319: volume dist >320: type debug/io-stats >321: option latency-measurement off >322: option count-fop-hits off >323: subvolumes dist-write-behind >324: end-volume >325: >326: volume test0-client-0 >327: type protocol/client >328: option ping-timeout 42 >329: option remote-host 10.70.35.198 >330: option remote-subvolume /brick0 >331: option transport-type socket >332: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >333: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >334: option send-gids true >335: end-volume >336: >337: volume test0-client-1 >338: type protocol/client >339: option ping-timeout 42 >340: option remote-host 10.70.35.240 >341: option remote-subvolume /brick0 >342: option transport-type socket >343: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >344: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >345: option send-gids true >346: end-volume >347: >348: volume test0-client-2 >349: type protocol/client >350: option ping-timeout 42 >351: option remote-host 10.70.35.172 >352: option remote-subvolume /brick0 >353: option transport-type socket >354: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >355: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >356: option send-gids true >357: end-volume >358: >359: volume test0-dht >360: type cluster/distribute >361: subvolumes test0-client-0 test0-client-1 test0-client-2 >362: end-volume >363: >364: volume test0-write-behind >365: type performance/write-behind >366: subvolumes test0-dht >367: end-volume >368: >369: volume test0 >370: type debug/io-stats >371: option latency-measurement off >372: option count-fop-hits off >373: subvolumes test0-write-behind >374: end-volume >375: >376: volume test1-client-0 >377: type protocol/client >378: option ping-timeout 42 >379: option remote-host 10.70.35.198 >380: option remote-subvolume /brick/1 >381: option transport-type socket >382: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >383: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >384: option send-gids true >385: end-volume >386: >387: volume test1-client-1 >388: type protocol/client >389: option ping-timeout 42 >390: option remote-host 10.70.35.198 >391: option remote-subvolume /brick/2 >392: option transport-type socket >393: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >394: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >395: option send-gids true >396: end-volume >397: >398: volume test1-client-2 >399: type protocol/client >400: option ping-timeout 42 >401: option remote-host 10.70.35.198 >402: option remote-subvolume /brick/3 >403: option transport-type socket >404: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >405: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >406: option send-gids true >407: end-volume >408: >409: volume test1-dht >410: type cluster/distribute >411: subvolumes test1-client-0 test1-client-1 test1-client-2 >412: end-volume >413: >414: volume test1-write-behind >415: type performance/write-behind >416: subvolumes test1-dht >417: end-volume >418: >419: volume test1 >420: type debug/io-stats >421: option latency-measurement off >422: option count-fop-hits off >423: subvolumes test1-write-behind >424: end-volume >425: >426: volume nfs-server >427: type nfs/server >428: option rpc-auth.auth-glusterfs on >429: option rpc-auth.auth-unix on >430: option rpc-auth.auth-null on >431: option rpc-auth.ports.insecure on >432: option rpc-auth-allow-insecure on >433: option transport-type socket >434: option transport.socket.listen-port 2049 >435: option nfs.dynamic-volumes on >436: option nfs.nlm on >437: option nfs.drc off >438: option rpc-auth.addr.dist.allow * >439: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >440: option rpc-auth.addr.test0.allow * >441: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >442: option rpc-auth.addr.test1.allow * >443: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >444: subvolumes dist test0 test1 >445: end-volume >446: >+------------------------------------------------------------------------------+ >[2014-08-21 08:29:46.461726] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:29:46.461872] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:29:46.461986] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:29:46.462078] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:29:46.462171] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:29:46.462300] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:29:46.462390] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:29:46.462523] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:29:46.462627] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:29:46.462715] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:29:46.462801] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:29:46.513128] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 08:29:46.513260] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 08:29:46.513349] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:29:46.513450] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:29:46.513537] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:29:46.513621] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:29:46.542052] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.542173] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.542303] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.542528] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.542635] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.542769] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.543008] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.543281] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.543388] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.544407] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:29:46.544455] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.544637] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:29:46.544659] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.544779] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:29:46.544807] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.544961] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:29:46.544991] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.545218] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:29:46.545242] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.545370] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:29:46.545390] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.545753] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:29:46.545784] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.546128] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:29:46.546152] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.546332] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:29:46.546362] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.546558] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.546765] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.546872] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.547012] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.547341] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.547454] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.547546] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.548005] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:29:46.548038] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:29:46.548059] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:29:46.548079] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:29:46.548099] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:29:46.548119] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:29:46.548149] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:29:46.548170] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:29:46.548225] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:29:46.548260] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 08:29:46.548275] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.548447] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 08:29:46.548476] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.548618] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:29:46.548647] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.549053] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:29:46.549083] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.549448] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:29:46.549473] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.549802] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:29:46.549824] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.549956] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:29:46.549976] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.550063] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:46.550308] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 08:29:46.550349] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 08:29:46.550388] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:29:46.550444] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:29:46.550485] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:29:46.550522] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:29:46.550764] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:29:46.550806] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:29:46.550821] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:46.551359] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:29:47.375364] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:29:47.375486] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:29:47.375562] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:29:47.389914] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:29:47.389977] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:29:47.390013] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 08:29:47.390061] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 08:29:47.390112] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:29:47.417539] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:29:47.417627] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:29:47.417690] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:29:47.417762] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:29:47.417839] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:29:47.417889] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:29:47.417943] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 08:29:47.417977] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 08:29:47.418008] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:29:47.418191] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.418318] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.465375] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.465519] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.465603] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.465697] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:29:47.465715] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.465844] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:29:47.465863] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.465973] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.466057] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.466127] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.466605] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:29:47.466635] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:29:47.466666] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:29:47.466680] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.466792] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 08:29:47.466811] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.466923] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.466995] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.467083] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:29:47.467103] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.467217] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:29:47.467251] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.467361] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:29:47.467381] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.467547] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:29:47.467575] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 08:29:47.467605] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:29:47.467619] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.467729] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:29:47.467747] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.467848] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:29:47.467877] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.467945] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468015] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468093] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468177] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468261] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468344] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:29:47.468496] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:29:47.468525] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:29:47.468562] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:29:47.468577] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.468691] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 08:29:47.468710] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.468816] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:29:47.468844] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:29:47.468972] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:29:47.469021] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:29:47.469036] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.469154] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:29:47.469175] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.469502] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:29:47.469530] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.469881] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:29:47.469905] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.470138] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:29:47.470167] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:29:47.470188] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:29:47.470367] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 08:29:47.471143] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:29:47.471460] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 08:29:47.471484] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.471809] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 08:29:47.472684] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 08:29:47.472724] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:29:47.473251] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 08:31:13.744528] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:31:13.757107] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:31:15.019531] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:31:15.041026] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:31:15.155478] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:31:15.155510] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:31:15.155718] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:31:15.170596] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:31:15.172907] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:31:15.175130] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:31:15.175253] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:31:15.175338] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:31:15.176090] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.181129] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.186047] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.190275] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.195234] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.200059] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.204897] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.209720] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.214541] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.219387] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.224197] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.229042] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.233897] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.238707] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.243468] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.248273] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.253089] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.257860] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.262649] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.267606] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.272320] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.277020] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.281559] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.285914] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.290103] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.294542] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.298953] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.303355] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.307598] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.312015] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.316692] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.320944] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.325397] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.329804] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.334061] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.338157] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:31:15.342614] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-dht >343: type cluster/distribute >344: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 >345: end-volume >346: >347: volume dist-write-behind >348: type performance/write-behind >349: subvolumes dist-dht >350: end-volume >351: >352: volume dist >353: type debug/io-stats >354: option latency-measurement off >355: option count-fop-hits off >356: subvolumes dist-write-behind >357: end-volume >358: >359: volume test0-client-0 >360: type protocol/client >361: option ping-timeout 42 >362: option remote-host 10.70.35.198 >363: option remote-subvolume /brick0 >364: option transport-type socket >365: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >366: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >367: option send-gids true >368: end-volume >369: >370: volume test0-client-1 >371: type protocol/client >372: option ping-timeout 42 >373: option remote-host 10.70.35.240 >374: option remote-subvolume /brick0 >375: option transport-type socket >376: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >377: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >378: option send-gids true >379: end-volume >380: >381: volume test0-client-2 >382: type protocol/client >383: option ping-timeout 42 >384: option remote-host 10.70.35.172 >385: option remote-subvolume /brick0 >386: option transport-type socket >387: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >388: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >389: option send-gids true >390: end-volume >391: >392: volume test0-dht >393: type cluster/distribute >394: subvolumes test0-client-0 test0-client-1 test0-client-2 >395: end-volume >396: >397: volume test0-write-behind >398: type performance/write-behind >399: subvolumes test0-dht >400: end-volume >401: >402: volume test0 >403: type debug/io-stats >404: option latency-measurement off >405: option count-fop-hits off >406: subvolumes test0-write-behind >407: end-volume >408: >409: volume test1-client-0 >410: type protocol/client >411: option ping-timeout 42 >412: option remote-host 10.70.35.198 >413: option remote-subvolume /brick/1 >414: option transport-type socket >415: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >416: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >417: option send-gids true >418: end-volume >419: >420: volume test1-client-1 >421: type protocol/client >422: option ping-timeout 42 >423: option remote-host 10.70.35.198 >424: option remote-subvolume /brick/2 >425: option transport-type socket >426: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >427: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >428: option send-gids true >429: end-volume >430: >431: volume test1-client-2 >432: type protocol/client >433: option ping-timeout 42 >434: option remote-host 10.70.35.198 >435: option remote-subvolume /brick/3 >436: option transport-type socket >437: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >438: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >439: option send-gids true >440: end-volume >441: >442: volume test1-dht >443: type cluster/distribute >444: subvolumes test1-client-0 test1-client-1 test1-client-2 >445: end-volume >446: >447: volume test1-write-behind >448: type performance/write-behind >449: subvolumes test1-dht >450: end-volume >451: >452: volume test1 >453: type debug/io-stats >454: option latency-measurement off >455: option count-fop-hits off >456: subvolumes test1-write-behind >457: end-volume >458: >459: volume nfs-server >460: type nfs/server >461: option rpc-auth.auth-glusterfs on >462: option rpc-auth.auth-unix on >463: option rpc-auth.auth-null on >464: option rpc-auth.ports.insecure on >465: option rpc-auth-allow-insecure on >466: option transport-type socket >467: option transport.socket.listen-port 2049 >468: option nfs.dynamic-volumes on >469: option nfs.nlm on >470: option nfs.drc off >471: option rpc-auth.addr.dist.allow * >472: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >473: option rpc-auth.addr.test0.allow * >474: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >475: option rpc-auth.addr.test1.allow * >476: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >477: subvolumes dist test0 test1 >478: end-volume >479: >+------------------------------------------------------------------------------+ >[2014-08-21 08:31:15.355231] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:31:15.355340] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:31:15.355462] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:31:15.355574] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:31:15.355662] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:31:15.355750] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:31:15.355835] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:31:15.355922] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:31:15.356008] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:31:15.356095] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:31:15.356180] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:31:15.356288] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 08:31:15.356375] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 08:31:15.356510] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 08:31:15.356599] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:31:15.356694] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:31:15.356781] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:31:15.356866] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:31:15.432733] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.432830] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.432892] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.432948] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433002] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433068] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433120] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433179] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433617] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433676] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433885] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.433949] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.434006] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.434226] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.434575] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.434810] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.434901] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.435140] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:31:15.435159] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.435277] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:31:15.435295] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.435668] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:31:15.435756] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.435884] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:31:15.435909] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436035] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:31:15.436055] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436169] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:31:15.436188] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436320] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:31:15.436340] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436500] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:31:15.436519] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436627] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:31:15.436647] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.436739] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:31:15.436758] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.437011] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:31:15.437030] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.437293] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 08:31:15.437315] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.437465] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 08:31:15.437490] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.437757] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 08:31:15.437781] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.438267] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:31:15.438289] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.438613] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:31:15.438633] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.438724] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:31:15.438750] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.439040] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:15.439129] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:31:15.439162] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:31:15.439180] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:31:15.439197] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:31:15.439226] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:31:15.439247] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:31:15.439270] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:31:15.439290] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:31:15.439307] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:31:15.439325] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:31:15.439342] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:31:15.439359] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 08:31:15.439376] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 08:31:15.439393] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 08:31:15.439410] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:31:15.439426] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:31:15.439481] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:31:15.439623] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:31:15.439639] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:15.440096] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:31:16.316809] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:31:16.321042] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:31:16.321100] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:31:16.321146] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:31:16.321194] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:31:16.321234] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 08:31:16.321290] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 08:31:16.321330] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 08:31:16.321367] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:31:16.359258] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:31:16.359377] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:31:16.359485] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:31:16.359548] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:31:16.359621] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:31:16.359697] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:31:16.359752] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 08:31:16.359812] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 08:31:16.359862] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 08:31:16.359903] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:31:16.360052] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.360115] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.360180] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.360243] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.403257] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.403393] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:31:16.403411] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.403593] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:31:16.403634] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.403739] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:31:16.403757] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.403855] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:31:16.403872] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.403953] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.404022] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.404092] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.404520] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:31:16.404550] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.404661] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:31:16.404687] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.404879] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.404965] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405069] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405140] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405203] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405316] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405426] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405567] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405674] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405776] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:31:16.405891] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:31:16.405916] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.406084] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:31:16.406122] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.406300] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 08:31:16.406328] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.406510] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:31:16.406536] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.406708] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:31:16.406749] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 08:31:16.406798] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:31:16.406822] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.406983] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 08:31:16.407010] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.407165] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:31:16.407191] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.407366] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:31:16.407417] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:31:16.407482] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.407625] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:31:16.407667] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:31:16.407717] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:31:16.407741] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.407954] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:31:16.408031] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 08:31:16.408071] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:31:16.408135] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 08:31:16.408156] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.408309] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 08:31:16.408328] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.408415] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:31:16.408482] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:31:16.408505] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:31:16.408622] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:31:16.408743] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 08:31:16.408800] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 08:31:16.408963] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:31:16.409122] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:31:16.409161] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.409531] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:31:16.409736] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 08:31:16.409757] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.410103] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 08:31:16.410672] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 08:31:16.410694] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:31:16.411204] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 08:49:04.987263] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:49:04.995302] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:49:06.128398] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:49:06.142187] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:49:06.180113] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:49:06.180157] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:49:06.180386] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:49:06.193662] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:49:06.195972] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:49:06.197683] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:49:06.197766] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:49:06.197828] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:49:06.198356] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.202026] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.206141] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.209705] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.213575] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.217350] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.221022] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.225095] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.229369] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.233632] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.237877] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.241812] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.245591] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.249100] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.252706] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.256632] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.260458] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.264166] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.267653] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.271193] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.274747] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.278672] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.282314] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.286033] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.290030] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.294128] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.297896] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.301584] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.305832] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.309517] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.313291] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.316959] I [client.c:2280:notify] 0-dist-client-31: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.320827] I [client.c:2280:notify] 0-dist-client-32: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.324464] I [client.c:2280:notify] 0-dist-client-33: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.328154] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.331944] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.335561] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.339210] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.342892] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:49:06.346629] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-client-31 >343: type protocol/client >344: option ping-timeout 42 >345: option remote-host 10.70.35.198 >346: option remote-subvolume /brick2/dist33 >347: option transport-type socket >348: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >349: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >350: option send-gids true >351: end-volume >352: >353: volume dist-client-32 >354: type protocol/client >355: option ping-timeout 42 >356: option remote-host 10.70.35.240 >357: option remote-subvolume /brick2/dist32 >358: option transport-type socket >359: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >360: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >361: option send-gids true >362: end-volume >363: >364: volume dist-client-33 >365: type protocol/client >366: option ping-timeout 42 >367: option remote-host 10.70.35.172 >368: option remote-subvolume /brick2/dist31 >369: option transport-type socket >370: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >371: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >372: option send-gids true >373: end-volume >374: >375: volume dist-dht >376: type cluster/distribute >377: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 dist-client-31 dist-client-32 dist-client-33 >378: end-volume >379: >380: volume dist-write-behind >381: type performance/write-behind >382: subvolumes dist-dht >383: end-volume >384: >385: volume dist >386: type debug/io-stats >387: option latency-measurement off >388: option count-fop-hits off >389: subvolumes dist-write-behind >390: end-volume >391: >392: volume test0-client-0 >393: type protocol/client >394: option ping-timeout 42 >395: option remote-host 10.70.35.198 >396: option remote-subvolume /brick0 >397: option transport-type socket >398: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >399: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >400: option send-gids true >401: end-volume >402: >403: volume test0-client-1 >404: type protocol/client >405: option ping-timeout 42 >406: option remote-host 10.70.35.240 >407: option remote-subvolume /brick0 >408: option transport-type socket >409: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >410: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >411: option send-gids true >412: end-volume >413: >414: volume test0-client-2 >415: type protocol/client >416: option ping-timeout 42 >417: option remote-host 10.70.35.172 >418: option remote-subvolume /brick0 >419: option transport-type socket >420: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >421: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >422: option send-gids true >423: end-volume >424: >425: volume test0-dht >426: type cluster/distribute >427: subvolumes test0-client-0 test0-client-1 test0-client-2 >428: end-volume >429: >430: volume test0-write-behind >431: type performance/write-behind >432: subvolumes test0-dht >433: end-volume >434: >435: volume test0 >436: type debug/io-stats >437: option latency-measurement off >438: option count-fop-hits off >439: subvolumes test0-write-behind >440: end-volume >441: >442: volume test1-client-0 >443: type protocol/client >444: option ping-timeout 42 >445: option remote-host 10.70.35.198 >446: option remote-subvolume /brick/1 >447: option transport-type socket >448: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >449: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >450: option send-gids true >451: end-volume >452: >453: volume test1-client-1 >454: type protocol/client >455: option ping-timeout 42 >456: option remote-host 10.70.35.198 >457: option remote-subvolume /brick/2 >458: option transport-type socket >459: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >460: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >461: option send-gids true >462: end-volume >463: >464: volume test1-client-2 >465: type protocol/client >466: option ping-timeout 42 >467: option remote-host 10.70.35.198 >468: option remote-subvolume /brick/3 >469: option transport-type socket >470: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >471: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >472: option send-gids true >473: end-volume >474: >475: volume test1-dht >476: type cluster/distribute >477: subvolumes test1-client-0 test1-client-1 test1-client-2 >478: end-volume >479: >480: volume test1-write-behind >481: type performance/write-behind >482: subvolumes test1-dht >483: end-volume >484: >485: volume test1 >486: type debug/io-stats >487: option latency-measurement off >488: option count-fop-hits off >489: subvolumes test1-write-behind >490: end-volume >491: >492: volume nfs-server >493: type nfs/server >494: option rpc-auth.auth-glusterfs on >495: option rpc-auth.auth-unix on >496: option rpc-auth.auth-null on >497: option rpc-auth.ports.insecure on >498: option rpc-auth-allow-insecure on >499: option transport-type socket >500: option transport.socket.listen-port 2049 >501: option nfs.dynamic-volumes on >502: option nfs.nlm on >503: option nfs.drc off >504: option rpc-auth.addr.dist.allow * >505: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >506: option rpc-auth.addr.test0.allow * >507: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >508: option rpc-auth.addr.test1.allow * >509: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >510: subvolumes dist test0 test1 >511: end-volume >512: >+------------------------------------------------------------------------------+ >[2014-08-21 08:49:06.355716] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:49:06.355764] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:49:06.355797] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:49:06.355828] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:49:06.355859] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:49:06.355890] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:49:06.355921] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:49:06.355953] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:49:06.355993] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:49:06.356081] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:49:06.356160] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:49:06.356242] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 08:49:06.356363] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 08:49:06.356468] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 08:49:06.356549] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-31: changing port to 49206 (from 0) >[2014-08-21 08:49:06.417035] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:49:06.417148] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:49:06.417239] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:49:06.417346] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:49:06.436523] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.436611] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.436673] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.436731] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.436811] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437043] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437145] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437221] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437495] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437597] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437692] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437765] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.437832] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.438056] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.438504] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:49:06.438528] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.438676] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:49:06.438706] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.438804] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:49:06.438823] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.438959] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:49:06.438979] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.439071] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:49:06.439091] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.439192] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:49:06.439212] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.439306] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:49:06.439339] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.439647] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:49:06.439669] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.440280] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:49:06.440305] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.440412] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:49:06.440459] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.440587] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:49:06.440607] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.440728] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 08:49:06.440749] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.440841] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 08:49:06.440860] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.441115] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 08:49:06.441137] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.441264] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.441361] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.441629] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.441744] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-31: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.441825] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:49:06.441852] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:49:06.441873] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:49:06.441892] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:49:06.441912] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:49:06.441932] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:49:06.441952] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:49:06.441973] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:49:06.441992] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:49:06.442012] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:49:06.442032] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:49:06.442054] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 08:49:06.442074] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 08:49:06.442094] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 08:49:06.442122] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:49:06.442136] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.442256] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:49:06.442277] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.442601] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:06.442798] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:49:06.442828] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:49:06.442857] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:49:06.442871] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.442994] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-31: Connected to dist-client-31, attached to remote volume '/brick2/dist33'. >[2014-08-21 08:49:06.443014] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-31: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.443397] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:49:06.443463] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:49:06.443482] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:06.443768] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:49:06.443798] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-31: Server lk version = 1 >[2014-08-21 08:49:07.406480] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:49:07.406556] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:49:07.413948] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:49:07.414014] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:49:07.421641] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:49:07.421690] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 08:49:07.421725] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 08:49:07.421752] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 08:49:07.421787] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-33: changing port to 49185 (from 0) >[2014-08-21 08:49:07.421823] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:49:07.442661] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:49:07.443139] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.443211] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.446961] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:49:07.447028] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:49:07.447074] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.447386] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:49:07.447404] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.447531] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:49:07.447548] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.454829] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:49:07.454915] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:49:07.454947] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:49:07.454983] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 08:49:07.455019] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 08:49:07.455056] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 08:49:07.455107] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-32: changing port to 49184 (from 0) >[2014-08-21 08:49:07.455158] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.455238] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:49:07.455344] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:49:07.455372] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:49:07.485238] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.485366] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.485463] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-33: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.485541] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.485613] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:49:07.485633] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.485728] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.485791] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.486102] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 08:49:07.486120] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.486197] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:49:07.486212] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.486293] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.486376] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:49:07.486393] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.486790] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 08:49:07.486819] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:49:07.486838] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:49:07.486857] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:49:07.486878] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.486959] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 08:49:07.486976] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.487082] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487155] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 08:49:07.487174] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.487286] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:49:07.487302] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.487384] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 08:49:07.487413] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:49:07.487424] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.487534] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487595] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487656] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487720] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487778] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487845] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-32: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487911] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.487974] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:49:07.488127] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:49:07.488158] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:49:07.488186] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:49:07.488200] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488295] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 08:49:07.488311] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488404] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 08:49:07.488419] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488528] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 08:49:07.488569] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:49:07.488583] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488677] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:49:07.488693] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488827] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:49:07.488856] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 08:49:07.488874] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 08:49:07.488903] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:49:07.488915] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.488999] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:49:07.489014] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.489194] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 08:49:07.489212] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.489406] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:49:07.489458] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 08:49:07.489481] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:49:07.489499] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:49:07.489736] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 08:49:07.489989] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:49:07.490009] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.490108] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-33: Connected to dist-client-33, attached to remote volume '/brick2/dist31'. >[2014-08-21 08:49:07.490125] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-33: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.490260] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:49:07.490354] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-32: Connected to dist-client-32, attached to remote volume '/brick2/dist32'. >[2014-08-21 08:49:07.490372] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-32: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:49:07.490563] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-33: Server lk version = 1 >[2014-08-21 08:49:07.490647] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-32: Server lk version = 1 >[2014-08-21 08:58:12.082804] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 08:58:12.088924] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 08:58:13.206840] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 08:58:13.222000] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 08:58:13.262946] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 08:58:13.262971] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 08:58:13.263157] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:58:13.273707] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:58:13.275484] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 08:58:13.276693] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 08:58:13.276749] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 08:58:13.276788] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 08:58:13.277279] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.280190] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.282305] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.284538] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.286934] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.289478] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.291682] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.293751] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.296038] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.298354] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.300410] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.302601] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.304767] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.306912] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.309228] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.311289] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.313338] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.315529] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.317810] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.319979] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.322245] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.324415] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.326703] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.328803] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.330837] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.333052] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.335501] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.337688] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.339749] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.341948] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.343999] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.346214] I [client.c:2280:notify] 0-dist-client-31: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.348424] I [client.c:2280:notify] 0-dist-client-32: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.350473] I [client.c:2280:notify] 0-dist-client-33: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.352530] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.354610] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.356932] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.358983] I [client.c:2280:notify] 0-test0-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.361091] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.363179] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 08:58:13.365415] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-client-31 >343: type protocol/client >344: option ping-timeout 42 >345: option remote-host 10.70.35.198 >346: option remote-subvolume /brick2/dist33 >347: option transport-type socket >348: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >349: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >350: option send-gids true >351: end-volume >352: >353: volume dist-client-32 >354: type protocol/client >355: option ping-timeout 42 >356: option remote-host 10.70.35.240 >357: option remote-subvolume /brick2/dist32 >358: option transport-type socket >359: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >360: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >361: option send-gids true >362: end-volume >363: >364: volume dist-client-33 >365: type protocol/client >366: option ping-timeout 42 >367: option remote-host 10.70.35.172 >368: option remote-subvolume /brick2/dist31 >369: option transport-type socket >370: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >371: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >372: option send-gids true >373: end-volume >374: >375: volume dist-dht >376: type cluster/distribute >377: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 dist-client-31 dist-client-32 dist-client-33 >378: end-volume >379: >380: volume dist-write-behind >381: type performance/write-behind >382: subvolumes dist-dht >383: end-volume >384: >385: volume dist >386: type debug/io-stats >387: option latency-measurement off >388: option count-fop-hits off >389: subvolumes dist-write-behind >390: end-volume >391: >392: volume test0-client-0 >393: type protocol/client >394: option ping-timeout 42 >395: option remote-host 10.70.35.198 >396: option remote-subvolume /brick0 >397: option transport-type socket >398: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >399: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >400: option send-gids true >401: end-volume >402: >403: volume test0-client-1 >404: type protocol/client >405: option ping-timeout 42 >406: option remote-host 10.70.35.240 >407: option remote-subvolume /brick0 >408: option transport-type socket >409: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >410: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >411: option send-gids true >412: end-volume >413: >414: volume test0-client-2 >415: type protocol/client >416: option ping-timeout 42 >417: option remote-host 10.70.35.172 >418: option remote-subvolume /brick0 >419: option transport-type socket >420: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >421: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >422: option send-gids true >423: end-volume >424: >425: volume test0-client-3 >426: type protocol/client >427: option ping-timeout 42 >428: option remote-host 10.70.35.198 >429: option remote-subvolume /brick1 >430: option transport-type socket >431: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >432: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >433: option send-gids true >434: end-volume >435: >436: volume test0-dht >437: type cluster/distribute >438: subvolumes test0-client-0 test0-client-1 test0-client-2 test0-client-3 >439: end-volume >440: >441: volume test0-write-behind >442: type performance/write-behind >443: subvolumes test0-dht >444: end-volume >445: >446: volume test0 >447: type debug/io-stats >448: option latency-measurement off >449: option count-fop-hits off >450: subvolumes test0-write-behind >451: end-volume >452: >453: volume test1-client-0 >454: type protocol/client >455: option ping-timeout 42 >456: option remote-host 10.70.35.198 >457: option remote-subvolume /brick/1 >458: option transport-type socket >459: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >460: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >461: option send-gids true >462: end-volume >463: >464: volume test1-client-1 >465: type protocol/client >466: option ping-timeout 42 >467: option remote-host 10.70.35.198 >468: option remote-subvolume /brick/2 >469: option transport-type socket >470: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >471: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >472: option send-gids true >473: end-volume >474: >475: volume test1-client-2 >476: type protocol/client >477: option ping-timeout 42 >478: option remote-host 10.70.35.198 >479: option remote-subvolume /brick/3 >480: option transport-type socket >481: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >482: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >483: option send-gids true >484: end-volume >485: >486: volume test1-dht >487: type cluster/distribute >488: subvolumes test1-client-0 test1-client-1 test1-client-2 >489: end-volume >490: >491: volume test1-write-behind >492: type performance/write-behind >493: subvolumes test1-dht >494: end-volume >495: >496: volume test1 >497: type debug/io-stats >498: option latency-measurement off >499: option count-fop-hits off >500: subvolumes test1-write-behind >501: end-volume >502: >503: volume nfs-server >504: type nfs/server >505: option rpc-auth.auth-glusterfs on >506: option rpc-auth.auth-unix on >507: option rpc-auth.auth-null on >508: option rpc-auth.ports.insecure on >509: option rpc-auth-allow-insecure on >510: option transport-type socket >511: option transport.socket.listen-port 2049 >512: option nfs.dynamic-volumes on >513: option nfs.nlm on >514: option nfs.drc off >515: option rpc-auth.addr.dist.allow * >516: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >517: option rpc-auth.addr.test0.allow * >518: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >519: option rpc-auth.addr.test1.allow * >520: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >521: subvolumes dist test0 test1 >522: end-volume >523: >+------------------------------------------------------------------------------+ >[2014-08-21 08:58:13.372639] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 08:58:13.372707] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 08:58:13.372728] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 08:58:13.372753] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 08:58:13.372801] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 08:58:13.372850] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 08:58:13.372899] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 08:58:13.372948] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 08:58:13.372997] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 08:58:13.373045] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 08:58:13.373100] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 08:58:13.373151] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 08:58:13.373180] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 08:58:13.373206] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 08:58:13.373258] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 08:58:13.373279] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 08:58:13.373303] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 08:58:13.373351] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 08:58:13.373374] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 08:58:13.373407] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 08:58:13.373457] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 08:58:13.373484] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 08:58:13.373511] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 08:58:13.373560] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 08:58:13.373600] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 08:58:13.373626] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 08:58:13.373676] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 08:58:13.373708] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 08:58:13.373730] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 08:58:13.373779] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 08:58:13.373809] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 08:58:13.373832] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-31: changing port to 49206 (from 0) >[2014-08-21 08:58:13.373883] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-32: changing port to 49184 (from 0) >[2014-08-21 08:58:13.373914] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-33: changing port to 49185 (from 0) >[2014-08-21 08:58:13.373939] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 08:58:13.373989] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 08:58:13.374021] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-3: changing port to 49207 (from 0) >[2014-08-21 08:58:13.374091] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 08:58:13.374144] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 08:58:13.374196] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 08:58:13.458758] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 08:58:13.462278] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462337] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462381] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462424] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462480] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462520] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462559] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462597] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462740] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462783] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462827] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462866] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462908] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.462964] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463005] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463045] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463109] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463161] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463203] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463243] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463295] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463338] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463381] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463434] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463467] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463506] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463553] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463599] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463640] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463681] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463721] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463760] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-31: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463814] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-32: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463859] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-33: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463899] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463954] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.463997] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.464161] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.464307] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.464518] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 08:58:13.464529] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464613] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 08:58:13.464624] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464668] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 08:58:13.464678] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464737] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 08:58:13.464748] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464817] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 08:58:13.464828] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464882] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 08:58:13.464892] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.464962] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 08:58:13.464973] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465204] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 08:58:13.465217] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465359] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 08:58:13.465371] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465443] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 08:58:13.465454] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465517] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 08:58:13.465528] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465581] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 08:58:13.465597] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465660] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 08:58:13.465670] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465728] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 08:58:13.465739] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465792] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 08:58:13.465802] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465871] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 08:58:13.465881] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.465943] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 08:58:13.465953] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466008] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 08:58:13.466018] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466101] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 08:58:13.466113] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466168] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 08:58:13.466178] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466248] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 08:58:13.466259] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466323] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 08:58:13.466334] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466404] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 08:58:13.466414] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466482] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 08:58:13.466492] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466743] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 08:58:13.466755] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466817] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 08:58:13.466828] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466895] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 08:58:13.466905] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.466968] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 08:58:13.466978] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467039] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 08:58:13.467050] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467144] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 08:58:13.467155] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467218] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 08:58:13.467228] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467288] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-31: Connected to dist-client-31, attached to remote volume '/brick2/dist33'. >[2014-08-21 08:58:13.467298] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-31: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467365] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-32: Connected to dist-client-32, attached to remote volume '/brick2/dist32'. >[2014-08-21 08:58:13.467376] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-32: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467445] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-33: Connected to dist-client-33, attached to remote volume '/brick2/dist31'. >[2014-08-21 08:58:13.467456] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-33: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467621] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 08:58:13.467633] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467855] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 08:58:13.467867] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.467945] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-3: Connected to test0-client-3, attached to remote volume '/brick1'. >[2014-08-21 08:58:13.467956] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.468140] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 08:58:13.468153] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.468321] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 08:58:13.468333] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.468515] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.468584] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 08:58:13.468601] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 08:58:13.468612] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 08:58:13.468623] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 08:58:13.468634] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 08:58:13.468644] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 08:58:13.468655] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 08:58:13.468666] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 08:58:13.468677] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 08:58:13.468688] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 08:58:13.468698] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 08:58:13.468709] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 08:58:13.468720] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 08:58:13.468731] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 08:58:13.468742] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 08:58:13.468753] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 08:58:13.468763] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 08:58:13.468774] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 08:58:13.468786] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 08:58:13.468797] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 08:58:13.468807] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 08:58:13.468818] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 08:58:13.468829] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 08:58:13.468839] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 08:58:13.468855] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 08:58:13.468866] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 08:58:13.468877] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 08:58:13.468887] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 08:58:13.468898] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 08:58:13.468909] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 08:58:13.468919] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 08:58:13.468930] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-31: Server lk version = 1 >[2014-08-21 08:58:13.468940] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-32: Server lk version = 1 >[2014-08-21 08:58:13.468951] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-33: Server lk version = 1 >[2014-08-21 08:58:13.468962] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 08:58:13.468972] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 08:58:13.468982] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-3: Server lk version = 1 >[2014-08-21 08:58:13.468993] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 08:58:13.469003] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 08:58:13.469015] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 08:58:13.469232] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 08:58:13.469244] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.469818] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 08:58:13.469841] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 08:58:13.469849] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 08:58:13.471723] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 09:14:05.556737] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 09:14:05.690122] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 09:14:10.717597] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:10.717783] I [dht-shared.c:269:dht_parse_decommissioned_bricks] 0-test0-dht: decommissioning subvolume test0-client-3 >[2014-08-21 09:14:10.717834] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:10.717997] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:10.718159] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 09:14:10.718288] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 09:14:49.755456] W [socket.c:529:__socket_rwv] 0-test0-client-3: readv on 10.70.35.198:49207 failed (No data available) >[2014-08-21 09:14:49.755531] I [client.c:2215:client_rpc_notify] 0-test0-client-3: disconnected from test0-client-3. Client process will keep trying to connect to glusterd until brick's port is available >[2014-08-21 09:14:49.761803] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 09:14:50.846735] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 09:14:50.875794] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 09:14:50.981668] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 09:14:50.981702] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 09:14:50.981909] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:50.995552] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:50.997663] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 09:14:50.999418] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 09:14:50.999510] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 09:14:50.999580] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 09:14:51.000172] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.004131] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.008786] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.012499] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.016267] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.020091] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.024229] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.027930] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.031671] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.035431] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.040045] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.043901] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.047574] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.051268] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.055154] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.059273] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.063020] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.066904] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.071604] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.075404] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.079162] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.090025] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.093688] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.097359] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.100995] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.104827] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.108518] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.112172] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.115806] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.119613] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.123918] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.127579] I [client.c:2280:notify] 0-dist-client-31: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.131261] I [client.c:2280:notify] 0-dist-client-32: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.135095] I [client.c:2280:notify] 0-dist-client-33: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.138741] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.142492] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.146155] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.149795] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.154141] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 09:14:51.157818] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-client-31 >343: type protocol/client >344: option ping-timeout 42 >345: option remote-host 10.70.35.198 >346: option remote-subvolume /brick2/dist33 >347: option transport-type socket >348: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >349: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >350: option send-gids true >351: end-volume >352: >353: volume dist-client-32 >354: type protocol/client >355: option ping-timeout 42 >356: option remote-host 10.70.35.240 >357: option remote-subvolume /brick2/dist32 >358: option transport-type socket >359: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >360: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >361: option send-gids true >362: end-volume >363: >364: volume dist-client-33 >365: type protocol/client >366: option ping-timeout 42 >367: option remote-host 10.70.35.172 >368: option remote-subvolume /brick2/dist31 >369: option transport-type socket >370: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >371: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >372: option send-gids true >373: end-volume >374: >375: volume dist-dht >376: type cluster/distribute >377: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 dist-client-31 dist-client-32 dist-client-33 >378: end-volume >379: >380: volume dist-write-behind >381: type performance/write-behind >382: subvolumes dist-dht >383: end-volume >384: >385: volume dist >386: type debug/io-stats >387: option latency-measurement off >388: option count-fop-hits off >389: subvolumes dist-write-behind >390: end-volume >391: >392: volume test0-client-0 >393: type protocol/client >394: option ping-timeout 42 >395: option remote-host 10.70.35.198 >396: option remote-subvolume /brick0 >397: option transport-type socket >398: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >399: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >400: option send-gids true >401: end-volume >402: >403: volume test0-client-1 >404: type protocol/client >405: option ping-timeout 42 >406: option remote-host 10.70.35.240 >407: option remote-subvolume /brick0 >408: option transport-type socket >409: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >410: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >411: option send-gids true >412: end-volume >413: >414: volume test0-client-2 >415: type protocol/client >416: option ping-timeout 42 >417: option remote-host 10.70.35.172 >418: option remote-subvolume /brick0 >419: option transport-type socket >420: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >421: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >422: option send-gids true >423: end-volume >424: >425: volume test0-dht >426: type cluster/distribute >427: subvolumes test0-client-0 test0-client-1 test0-client-2 >428: end-volume >429: >430: volume test0-write-behind >431: type performance/write-behind >432: subvolumes test0-dht >433: end-volume >434: >435: volume test0 >436: type debug/io-stats >437: option latency-measurement off >438: option count-fop-hits off >439: subvolumes test0-write-behind >440: end-volume >441: >442: volume test1-client-0 >443: type protocol/client >444: option ping-timeout 42 >445: option remote-host 10.70.35.198 >446: option remote-subvolume /brick/1 >447: option transport-type socket >448: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >449: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >450: option send-gids true >451: end-volume >452: >453: volume test1-client-1 >454: type protocol/client >455: option ping-timeout 42 >456: option remote-host 10.70.35.198 >457: option remote-subvolume /brick/2 >458: option transport-type socket >459: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >460: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >461: option send-gids true >462: end-volume >463: >464: volume test1-client-2 >465: type protocol/client >466: option ping-timeout 42 >467: option remote-host 10.70.35.198 >468: option remote-subvolume /brick/3 >469: option transport-type socket >470: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >471: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >472: option send-gids true >473: end-volume >474: >475: volume test1-dht >476: type cluster/distribute >477: subvolumes test1-client-0 test1-client-1 test1-client-2 >478: end-volume >479: >480: volume test1-write-behind >481: type performance/write-behind >482: subvolumes test1-dht >483: end-volume >484: >485: volume test1 >486: type debug/io-stats >487: option latency-measurement off >488: option count-fop-hits off >489: subvolumes test1-write-behind >490: end-volume >491: >492: volume nfs-server >493: type nfs/server >494: option rpc-auth.auth-glusterfs on >495: option rpc-auth.auth-unix on >496: option rpc-auth.auth-null on >497: option rpc-auth.ports.insecure on >498: option rpc-auth-allow-insecure on >499: option transport-type socket >500: option transport.socket.listen-port 2049 >501: option nfs.dynamic-volumes on >502: option nfs.nlm on >503: option nfs.drc off >504: option rpc-auth.addr.dist.allow * >505: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >506: option rpc-auth.addr.test0.allow * >507: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >508: option rpc-auth.addr.test1.allow * >509: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >510: subvolumes dist test0 test1 >511: end-volume >512: >+------------------------------------------------------------------------------+ >[2014-08-21 09:14:51.167624] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 09:14:51.167705] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 09:14:51.167750] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 09:14:51.167807] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 09:14:51.167842] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 09:14:51.167873] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 09:14:51.167903] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 09:14:51.167934] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 09:14:51.167966] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 09:14:51.167998] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 09:14:51.168028] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 09:14:51.168084] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 09:14:51.168122] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 09:14:51.168153] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 09:14:51.168183] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-31: changing port to 49206 (from 0) >[2014-08-21 09:14:51.168214] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 09:14:51.232178] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 09:14:51.232239] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 09:14:51.239850] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 09:14:51.244485] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.244593] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.244655] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.244893] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.244961] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245019] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245103] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245321] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245407] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245634] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245719] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245782] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.245985] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.246232] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.246325] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.246661] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 09:14:51.246681] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.246928] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 09:14:51.246948] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.247089] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 09:14:51.247108] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.247367] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 09:14:51.247386] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.247501] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 09:14:51.247519] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.247691] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 09:14:51.247710] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.247962] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 09:14:51.247981] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.248093] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 09:14:51.248112] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.248497] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 09:14:51.248517] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.248625] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 09:14:51.248643] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.248722] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 09:14:51.248740] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.249090] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 09:14:51.249112] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.249361] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 09:14:51.249381] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.249499] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 09:14:51.249518] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.249592] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-31: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.249913] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 09:14:51.249943] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 09:14:51.249963] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 09:14:51.249980] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 09:14:51.249997] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 09:14:51.250014] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 09:14:51.250032] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 09:14:51.250050] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 09:14:51.250067] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 09:14:51.250903] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 09:14:51.250924] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 09:14:51.250942] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 09:14:51.250960] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 09:14:51.250977] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 09:14:51.251002] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-31: Connected to dist-client-31, attached to remote volume '/brick2/dist33'. >[2014-08-21 09:14:51.251014] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-31: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.251216] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.251465] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.251568] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 09:14:51.251588] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.251832] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.252132] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-31: Server lk version = 1 >[2014-08-21 09:14:51.252166] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 09:14:51.252179] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.252463] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 09:14:51.252482] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.252593] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 09:14:51.252631] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 09:14:51.252645] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:51.253244] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 09:14:51.253272] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 09:14:51.253298] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 09:14:51.925213] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 09:14:51.930472] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 09:14:51.930558] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 09:14:51.930702] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 09:14:51.930791] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 09:14:51.930838] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 09:14:51.952510] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 09:14:51.952595] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 09:14:51.952630] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-33: changing port to 49185 (from 0) >[2014-08-21 09:14:51.952673] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 09:14:51.969126] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.969271] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 09:14:51.969322] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.969414] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.976398] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 09:14:51.976481] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 09:14:51.976558] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 09:14:51.976607] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 09:14:51.976652] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 09:14:51.976697] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.976804] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 09:14:51.977028] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 09:14:51.977250] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:51.977419] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 09:14:51.977523] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-32: changing port to 49184 (from 0) >[2014-08-21 09:14:51.977598] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 09:14:51.977693] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.004968] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 09:14:52.005011] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.026987] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.027100] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-33: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.027210] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.027297] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 09:14:52.027348] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.027444] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.027583] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 09:14:52.027602] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.027772] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 09:14:52.027874] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.028063] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 09:14:52.028097] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 09:14:52.028182] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.028351] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-33: Connected to dist-client-33, attached to remote volume '/brick2/dist31'. >[2014-08-21 09:14:52.028378] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-33: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.028615] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 09:14:52.028636] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.028760] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 09:14:52.028793] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 09:14:52.028806] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.028929] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 09:14:52.028964] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 09:14:52.028977] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.029104] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 09:14:52.029122] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.029881] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 09:14:52.029954] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-33: Server lk version = 1 >[2014-08-21 09:14:52.029976] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 09:14:52.030001] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 09:14:52.030026] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 09:14:52.030080] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.030186] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.030453] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.030605] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 09:14:52.030635] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.030861] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.030997] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031093] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031191] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031267] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 09:14:52.031292] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031409] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-32: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031477] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 09:14:52.031678] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 09:14:52.031696] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.031832] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 09:14:52.031850] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.031969] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 09:14:52.031986] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032084] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 09:14:52.032102] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032229] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 09:14:52.032247] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032382] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-32: Connected to dist-client-32, attached to remote volume '/brick2/dist32'. >[2014-08-21 09:14:52.032400] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-32: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032518] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 09:14:52.032535] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032632] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 09:14:52.032648] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.032767] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 09:14:52.032785] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.033244] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 09:14:52.033266] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.033473] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 09:14:52.033560] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 09:14:52.033590] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 09:14:52.033624] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 09:14:52.034054] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-32: Server lk version = 1 >[2014-08-21 09:14:52.034082] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 09:14:52.034100] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 09:14:52.034344] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 09:14:52.034396] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 09:14:52.034415] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 09:14:52.034433] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 09:14:52.034450] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 09:14:52.034467] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 10:22:28.387859] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 10:22:28.394238] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing >[2014-08-21 10:22:42.638469] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 10:22:42.664880] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 10:22:42.784440] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 10:22:42.784487] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 10:22:42.784721] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:22:42.799802] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:22:42.801856] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:22:42.803902] I [dht-shared.c:334:dht_init_regex] 0-workstation-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:22:42.805510] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 10:22:42.805619] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 10:22:42.805701] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 10:22:42.806463] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.810984] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.815265] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.819515] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.823792] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.828064] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.832416] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.836752] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.841030] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.845336] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.849580] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.853896] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.858135] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.862402] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.866671] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.870919] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.875163] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.879528] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.883852] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.888092] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.892399] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.896652] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.900897] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.905245] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.909506] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.913752] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.918034] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.922321] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.926562] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.930896] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.935154] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.939445] I [client.c:2280:notify] 0-dist-client-31: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.943742] I [client.c:2280:notify] 0-dist-client-32: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.947997] I [client.c:2280:notify] 0-dist-client-33: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.952276] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.956631] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.960900] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.965156] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.969475] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.973777] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.978076] I [client.c:2280:notify] 0-workstation-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:22:42.982433] I [client.c:2280:notify] 0-workstation-client-1: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-client-31 >343: type protocol/client >344: option ping-timeout 42 >345: option remote-host 10.70.35.198 >346: option remote-subvolume /brick2/dist33 >347: option transport-type socket >348: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >349: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >350: option send-gids true >351: end-volume >352: >353: volume dist-client-32 >354: type protocol/client >355: option ping-timeout 42 >356: option remote-host 10.70.35.240 >357: option remote-subvolume /brick2/dist32 >358: option transport-type socket >359: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >360: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >361: option send-gids true >362: end-volume >363: >364: volume dist-client-33 >365: type protocol/client >366: option ping-timeout 42 >367: option remote-host 10.70.35.172 >368: option remote-subvolume /brick2/dist31 >369: option transport-type socket >370: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >371: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >372: option send-gids true >373: end-volume >374: >375: volume dist-dht >376: type cluster/distribute >377: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 dist-client-31 dist-client-32 dist-client-33 >378: end-volume >379: >380: volume dist-write-behind >381: type performance/write-behind >382: subvolumes dist-dht >383: end-volume >384: >385: volume dist >386: type debug/io-stats >387: option latency-measurement off >388: option count-fop-hits off >389: subvolumes dist-write-behind >390: end-volume >391: >392: volume test0-client-0 >393: type protocol/client >394: option ping-timeout 42 >395: option remote-host 10.70.35.198 >396: option remote-subvolume /brick0 >397: option transport-type socket >398: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >399: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >400: option send-gids true >401: end-volume >402: >403: volume test0-client-1 >404: type protocol/client >405: option ping-timeout 42 >406: option remote-host 10.70.35.240 >407: option remote-subvolume /brick0 >408: option transport-type socket >409: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >410: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >411: option send-gids true >412: end-volume >413: >414: volume test0-client-2 >415: type protocol/client >416: option ping-timeout 42 >417: option remote-host 10.70.35.172 >418: option remote-subvolume /brick0 >419: option transport-type socket >420: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >421: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >422: option send-gids true >423: end-volume >424: >425: volume test0-dht >426: type cluster/distribute >427: subvolumes test0-client-0 test0-client-1 test0-client-2 >428: end-volume >429: >430: volume test0-write-behind >431: type performance/write-behind >432: subvolumes test0-dht >433: end-volume >434: >435: volume test0 >436: type debug/io-stats >437: option latency-measurement off >438: option count-fop-hits off >439: subvolumes test0-write-behind >440: end-volume >441: >442: volume test1-client-0 >443: type protocol/client >444: option ping-timeout 42 >445: option remote-host 10.70.35.198 >446: option remote-subvolume /brick/1 >447: option transport-type socket >448: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >449: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >450: option send-gids true >451: end-volume >452: >453: volume test1-client-1 >454: type protocol/client >455: option ping-timeout 42 >456: option remote-host 10.70.35.198 >457: option remote-subvolume /brick/2 >458: option transport-type socket >459: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >460: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >461: option send-gids true >462: end-volume >463: >464: volume test1-client-2 >465: type protocol/client >466: option ping-timeout 42 >467: option remote-host 10.70.35.198 >468: option remote-subvolume /brick/3 >469: option transport-type socket >470: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >471: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >472: option send-gids true >473: end-volume >474: >475: volume test1-dht >476: type cluster/distribute >477: subvolumes test1-client-0 test1-client-1 test1-client-2 >478: end-volume >479: >480: volume test1-write-behind >481: type performance/write-behind >482: subvolumes test1-dht >483: end-volume >484: >485: volume test1 >486: type debug/io-stats >487: option latency-measurement off >488: option count-fop-hits off >489: subvolumes test1-write-behind >490: end-volume >491: >492: volume workstation-client-0 >493: type protocol/client >494: option ping-timeout 42 >495: option remote-host 10.70.35.198 >496: option remote-subvolume /brick2/workstation0 >497: option transport-type socket >498: option username a7fd1896-75f7-4ea6-afcf-9eca43de8d10 >499: option password 78bbb91d-e14d-46c5-a3d1-f6a97d973c38 >500: option send-gids true >501: end-volume >502: >503: volume workstation-client-1 >504: type protocol/client >505: option ping-timeout 42 >506: option remote-host 10.70.35.240 >507: option remote-subvolume /brick2/workstation1 >508: option transport-type socket >509: option username a7fd1896-75f7-4ea6-afcf-9eca43de8d10 >510: option password 78bbb91d-e14d-46c5-a3d1-f6a97d973c38 >511: option send-gids true >512: end-volume >513: >514: volume workstation-dht >515: type cluster/distribute >516: subvolumes workstation-client-0 workstation-client-1 >517: end-volume >518: >519: volume workstation-write-behind >520: type performance/write-behind >521: subvolumes workstation-dht >522: end-volume >523: >524: volume workstation >525: type debug/io-stats >526: option latency-measurement off >527: option count-fop-hits off >528: subvolumes workstation-write-behind >529: end-volume >530: >531: volume nfs-server >532: type nfs/server >533: option rpc-auth.auth-glusterfs on >534: option rpc-auth.auth-unix on >535: option rpc-auth.auth-null on >536: option rpc-auth.ports.insecure on >537: option rpc-auth-allow-insecure on >538: option transport-type socket >539: option transport.socket.listen-port 2049 >540: option nfs.dynamic-volumes on >541: option nfs.nlm on >542: option nfs.drc off >543: option rpc-auth.addr.dist.allow * >544: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >545: option rpc-auth.addr.test0.allow * >546: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >547: option rpc-auth.addr.test1.allow * >548: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >549: option rpc-auth.addr.workstation.allow * >550: option nfs3.workstation.volume-id e5c0899d-d49a-485a-a821-23db66705470 >551: subvolumes dist test0 test1 workstation >552: end-volume >553: >+------------------------------------------------------------------------------+ >[2014-08-21 10:22:42.993666] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 10:22:42.993721] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 10:22:42.993760] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 10:22:42.993796] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 10:22:42.993863] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 10:22:42.993903] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 10:22:42.993940] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 10:22:42.994004] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 10:22:42.994044] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 10:22:42.994080] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 10:22:42.994116] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 10:22:42.994152] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 10:22:42.994216] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 10:22:42.994288] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 10:22:42.994343] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-31: changing port to 49206 (from 0) >[2014-08-21 10:22:42.994410] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 10:22:43.065235] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 10:22:43.065332] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 10:22:43.065395] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 10:22:43.078646] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-workstation-client-0: changing port to 49208 (from 0) >[2014-08-21 10:22:43.079513] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 10:22:43.079592] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 10:22:43.079639] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 10:22:43.079685] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 10:22:43.079730] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 10:22:43.079773] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 10:22:43.079806] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 10:22:43.079848] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 10:22:43.079880] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-33: changing port to 49185 (from 0) >[2014-08-21 10:22:43.079911] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 10:22:43.084794] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.084951] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085058] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085170] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085357] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085491] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085629] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085735] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085840] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.085974] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.086077] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.129431] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.129541] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.129783] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.130271] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 10:22:43.130303] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.130482] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 10:22:43.130503] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.130614] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 10:22:43.130633] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.130756] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 10:22:43.130785] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.130902] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 10:22:43.130922] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.131067] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 10:22:43.131096] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.131374] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 10:22:43.131405] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.131559] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 10:22:43.131588] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.131723] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 10:22:43.131753] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.132044] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 10:22:43.132067] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.132258] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 10:22:43.132287] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.132484] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 10:22:43.132506] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.132627] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 10:22:43.132647] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.132943] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 10:22:43.132971] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.133117] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.133269] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.133425] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.133703] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-31: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.133803] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.134449] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 10:22:43.134482] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 10:22:43.134504] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 10:22:43.134524] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 10:22:43.134544] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 10:22:43.134565] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 10:22:43.134593] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 10:22:43.134614] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 10:22:43.134634] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 10:22:43.134653] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 10:22:43.134673] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 10:22:43.134693] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 10:22:43.134713] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 10:22:43.134733] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 10:22:43.134761] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 10:22:43.134775] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.134928] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 10:22:43.134957] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.135252] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 10:22:43.135276] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.135858] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-31: Connected to dist-client-31, attached to remote volume '/brick2/dist33'. >[2014-08-21 10:22:43.135888] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-31: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.136073] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 10:22:43.136095] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.136238] I [client-handshake.c:1415:select_server_supported_programs] 0-workstation-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.136359] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.136653] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 10:22:43.136693] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 10:22:43.136730] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 10:22:43.136768] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-31: Server lk version = 1 >[2014-08-21 10:22:43.136805] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 10:22:43.136845] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.136966] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137083] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137244] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137387] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137500] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137621] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-33: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137738] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.137856] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.138394] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 10:22:43.138428] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.138611] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 10:22:43.138640] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.138813] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 10:22:43.138842] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.139016] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 10:22:43.139045] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.139395] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 10:22:43.139445] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 10:22:43.139486] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 10:22:43.139537] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 10:22:43.139565] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.139741] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 10:22:43.139770] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.139944] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 10:22:43.139973] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.140149] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 10:22:43.140218] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.140417] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-33: Connected to dist-client-33, attached to remote volume '/brick2/dist31'. >[2014-08-21 10:22:43.140456] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-33: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.140628] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 10:22:43.140658] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.141489] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 10:22:43.141541] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 10:22:43.141579] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 10:22:43.141616] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 10:22:43.141653] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-33: Server lk version = 1 >[2014-08-21 10:22:43.141690] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 10:22:43.141912] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 10:22:43.142212] I [client-handshake.c:1200:client_setvolume_cbk] 0-workstation-client-0: Connected to workstation-client-0, attached to remote volume '/brick2/workstation0'. >[2014-08-21 10:22:43.142236] I [client-handshake.c:1212:client_setvolume_cbk] 0-workstation-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.142447] I [client-handshake.c:188:client_set_lk_version_cbk] 0-workstation-client-0: Server lk version = 1 >[2014-08-21 10:22:43.731990] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 10:22:43.732083] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 10:22:43.732121] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 10:22:43.732168] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 10:22:43.732250] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 10:22:43.754112] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 10:22:43.754214] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 10:22:43.754265] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 10:22:43.754326] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 10:22:43.754364] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-32: changing port to 49184 (from 0) >[2014-08-21 10:22:43.754412] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 10:22:43.754446] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-workstation-client-1: changing port to 49185 (from 0) >[2014-08-21 10:22:43.786531] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.786628] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.786688] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.786759] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.786839] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.786927] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787000] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787064] I [client-handshake.c:1415:select_server_supported_programs] 0-workstation-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787136] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-32: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787237] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787339] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 10:22:43.787359] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.787473] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787556] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 10:22:43.787575] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.787686] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:22:43.787771] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 10:22:43.787789] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.787899] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 10:22:43.787918] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788028] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 10:22:43.788047] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788140] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 10:22:43.788160] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788296] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-32: Connected to dist-client-32, attached to remote volume '/brick2/dist32'. >[2014-08-21 10:22:43.788328] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-32: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788429] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 10:22:43.788448] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788531] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 10:22:43.788557] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.788645] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 10:22:43.788674] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-32: Server lk version = 1 >[2014-08-21 10:22:43.788723] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 10:22:43.788754] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 10:22:43.788769] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.789004] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 10:22:43.789043] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 10:22:43.789058] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.789468] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 10:22:43.789502] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 10:22:43.789525] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 10:22:43.789842] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 10:22:43.789875] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 10:22:43.791882] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 10:22:43.791929] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 10:22:43.792536] I [client-handshake.c:1200:client_setvolume_cbk] 0-workstation-client-1: Connected to workstation-client-1, attached to remote volume '/brick2/workstation1'. >[2014-08-21 10:22:43.792560] I [client-handshake.c:1212:client_setvolume_cbk] 0-workstation-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:22:43.793406] I [client-handshake.c:188:client_set_lk_version_cbk] 0-workstation-client-1: Server lk version = 1 >[2014-08-21 10:22:43.794102] I [dht-layout.c:663:dht_layout_normalize] 0-workstation-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 >[2014-08-21 10:23:06.914553] W [rpcsvc.c:261:rpcsvc_program_actor] 0-rpc-service: RPC program version not available (req 100003 4) >[2014-08-21 10:23:06.914598] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully >[2014-08-21 10:25:11.384498] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed >[2014-08-21 10:25:12.468389] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.0.27 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/1a2c771d5e33efe26254c34e210212a0.socket) >[2014-08-21 10:25:12.494000] I [rpcsvc.c:2131:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16 >[2014-08-21 10:25:12.558578] I [rpc-drc.c:731:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF >[2014-08-21 10:25:12.558615] I [nfs.c:1353:init] 0-nfs: NFS service started >[2014-08-21 10:25:12.558846] I [dht-shared.c:334:dht_init_regex] 0-dist-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:25:12.574307] I [dht-shared.c:334:dht_init_regex] 0-test0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:25:12.576431] I [dht-shared.c:334:dht_init_regex] 0-test1-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:25:12.578531] I [dht-shared.c:334:dht_init_regex] 0-workstation-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ >[2014-08-21 10:25:12.580600] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized >[2014-08-21 10:25:12.580712] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized >[2014-08-21 10:25:12.580794] W [graph.c:344:_log_if_unknown_option] 0-nfs-server: option 'transport-type' is not recognized >[2014-08-21 10:25:12.581565] I [client.c:2280:notify] 0-dist-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.587012] I [client.c:2280:notify] 0-dist-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.591363] I [client.c:2280:notify] 0-dist-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.595655] I [client.c:2280:notify] 0-dist-client-3: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.599955] I [client.c:2280:notify] 0-dist-client-4: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.604285] I [client.c:2280:notify] 0-dist-client-5: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.608602] I [client.c:2280:notify] 0-dist-client-6: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.612904] I [client.c:2280:notify] 0-dist-client-7: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.617229] I [client.c:2280:notify] 0-dist-client-8: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.621611] I [client.c:2280:notify] 0-dist-client-9: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.625888] I [client.c:2280:notify] 0-dist-client-10: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.630257] I [client.c:2280:notify] 0-dist-client-11: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.634562] I [client.c:2280:notify] 0-dist-client-12: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.638853] I [client.c:2280:notify] 0-dist-client-13: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.643196] I [client.c:2280:notify] 0-dist-client-14: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.647480] I [client.c:2280:notify] 0-dist-client-15: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.651769] I [client.c:2280:notify] 0-dist-client-16: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.656118] I [client.c:2280:notify] 0-dist-client-17: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.660402] I [client.c:2280:notify] 0-dist-client-18: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.664705] I [client.c:2280:notify] 0-dist-client-19: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.669032] I [client.c:2280:notify] 0-dist-client-20: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.673380] I [client.c:2280:notify] 0-dist-client-21: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.677711] I [client.c:2280:notify] 0-dist-client-22: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.682109] I [client.c:2280:notify] 0-dist-client-23: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.686394] I [client.c:2280:notify] 0-dist-client-24: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.690691] I [client.c:2280:notify] 0-dist-client-25: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.695022] I [client.c:2280:notify] 0-dist-client-26: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.699358] I [client.c:2280:notify] 0-dist-client-27: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.703663] I [client.c:2280:notify] 0-dist-client-28: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.707987] I [client.c:2280:notify] 0-dist-client-29: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.712339] I [client.c:2280:notify] 0-dist-client-30: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.716646] I [client.c:2280:notify] 0-dist-client-31: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.720979] I [client.c:2280:notify] 0-dist-client-32: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.725320] I [client.c:2280:notify] 0-dist-client-33: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.729629] I [client.c:2280:notify] 0-test0-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.734031] I [client.c:2280:notify] 0-test0-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.738349] I [client.c:2280:notify] 0-test0-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.742661] I [client.c:2280:notify] 0-test1-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.746991] I [client.c:2280:notify] 0-test1-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.751352] I [client.c:2280:notify] 0-test1-client-2: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.755722] I [client.c:2280:notify] 0-workstation-client-0: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.760088] I [client.c:2280:notify] 0-workstation-client-1: parent translators are ready, attempting connect on transport >[2014-08-21 10:25:12.764378] I [client.c:2280:notify] 0-workstation-client-2: parent translators are ready, attempting connect on transport >Final graph: >+------------------------------------------------------------------------------+ > 1: volume dist-client-0 > 2: type protocol/client > 3: option ping-timeout 42 > 4: option remote-host 10.70.35.198 > 5: option remote-subvolume /brick2/dist0 > 6: option transport-type socket > 7: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 8: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 9: option send-gids true > 10: end-volume > 11: > 12: volume dist-client-1 > 13: type protocol/client > 14: option ping-timeout 42 > 15: option remote-host 10.70.35.240 > 16: option remote-subvolume /brick2/dist1 > 17: option transport-type socket > 18: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 19: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 20: option send-gids true > 21: end-volume > 22: > 23: volume dist-client-2 > 24: type protocol/client > 25: option ping-timeout 42 > 26: option remote-host 10.70.35.172 > 27: option remote-subvolume /brick2/dist2 > 28: option transport-type socket > 29: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 30: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 31: option send-gids true > 32: end-volume > 33: > 34: volume dist-client-3 > 35: type protocol/client > 36: option ping-timeout 42 > 37: option remote-host 10.70.35.198 > 38: option remote-subvolume /brick2/dist3 > 39: option transport-type socket > 40: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 41: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 42: option send-gids true > 43: end-volume > 44: > 45: volume dist-client-4 > 46: type protocol/client > 47: option ping-timeout 42 > 48: option remote-host 10.70.35.198 > 49: option remote-subvolume /brick2/dist4 > 50: option transport-type socket > 51: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 52: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 53: option send-gids true > 54: end-volume > 55: > 56: volume dist-client-5 > 57: type protocol/client > 58: option ping-timeout 42 > 59: option remote-host 10.70.35.198 > 60: option remote-subvolume /brick2/dist5 > 61: option transport-type socket > 62: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 63: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 64: option send-gids true > 65: end-volume > 66: > 67: volume dist-client-6 > 68: type protocol/client > 69: option ping-timeout 42 > 70: option remote-host 10.70.35.198 > 71: option remote-subvolume /brick2/dist6 > 72: option transport-type socket > 73: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 74: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 75: option send-gids true > 76: end-volume > 77: > 78: volume dist-client-7 > 79: type protocol/client > 80: option ping-timeout 42 > 81: option remote-host 10.70.35.198 > 82: option remote-subvolume /brick2/dist7 > 83: option transport-type socket > 84: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 85: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 86: option send-gids true > 87: end-volume > 88: > 89: volume dist-client-8 > 90: type protocol/client > 91: option ping-timeout 42 > 92: option remote-host 10.70.35.198 > 93: option remote-subvolume /brick2/dist8 > 94: option transport-type socket > 95: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 > 96: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 > 97: option send-gids true > 98: end-volume > 99: >100: volume dist-client-9 >101: type protocol/client >102: option ping-timeout 42 >103: option remote-host 10.70.35.240 >104: option remote-subvolume /brick2/dist9 >105: option transport-type socket >106: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >107: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >108: option send-gids true >109: end-volume >110: >111: volume dist-client-10 >112: type protocol/client >113: option ping-timeout 42 >114: option remote-host 10.70.35.198 >115: option remote-subvolume /brick2/dist10 >116: option transport-type socket >117: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >118: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >119: option send-gids true >120: end-volume >121: >122: volume dist-client-11 >123: type protocol/client >124: option ping-timeout 42 >125: option remote-host 10.70.35.240 >126: option remote-subvolume /brick2/dist11 >127: option transport-type socket >128: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >129: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >130: option send-gids true >131: end-volume >132: >133: volume dist-client-12 >134: type protocol/client >135: option ping-timeout 42 >136: option remote-host 10.70.35.172 >137: option remote-subvolume /brick2/dist12 >138: option transport-type socket >139: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >140: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >141: option send-gids true >142: end-volume >143: >144: volume dist-client-13 >145: type protocol/client >146: option ping-timeout 42 >147: option remote-host 10.70.35.198 >148: option remote-subvolume /brick2/dist13 >149: option transport-type socket >150: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >151: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >152: option send-gids true >153: end-volume >154: >155: volume dist-client-14 >156: type protocol/client >157: option ping-timeout 42 >158: option remote-host 10.70.35.240 >159: option remote-subvolume /brick2/dist14 >160: option transport-type socket >161: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >162: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >163: option send-gids true >164: end-volume >165: >166: volume dist-client-15 >167: type protocol/client >168: option ping-timeout 42 >169: option remote-host 10.70.35.172 >170: option remote-subvolume /brick2/dist15 >171: option transport-type socket >172: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >173: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >174: option send-gids true >175: end-volume >176: >177: volume dist-client-16 >178: type protocol/client >179: option ping-timeout 42 >180: option remote-host 10.70.35.198 >181: option remote-subvolume /brick2/dist16 >182: option transport-type socket >183: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >184: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >185: option send-gids true >186: end-volume >187: >188: volume dist-client-17 >189: type protocol/client >190: option ping-timeout 42 >191: option remote-host 10.70.35.240 >192: option remote-subvolume /brick2/dist17 >193: option transport-type socket >194: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >195: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >196: option send-gids true >197: end-volume >198: >199: volume dist-client-18 >200: type protocol/client >201: option ping-timeout 42 >202: option remote-host 10.70.35.172 >203: option remote-subvolume /brick2/dist18 >204: option transport-type socket >205: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >206: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >207: option send-gids true >208: end-volume >209: >210: volume dist-client-19 >211: type protocol/client >212: option ping-timeout 42 >213: option remote-host 10.70.35.198 >214: option remote-subvolume /brick2/dist19 >215: option transport-type socket >216: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >217: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >218: option send-gids true >219: end-volume >220: >221: volume dist-client-20 >222: type protocol/client >223: option ping-timeout 42 >224: option remote-host 10.70.35.240 >225: option remote-subvolume /brick2/dist20 >226: option transport-type socket >227: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >228: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >229: option send-gids true >230: end-volume >231: >232: volume dist-client-21 >233: type protocol/client >234: option ping-timeout 42 >235: option remote-host 10.70.35.172 >236: option remote-subvolume /brick2/dist21 >237: option transport-type socket >238: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >239: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >240: option send-gids true >241: end-volume >242: >243: volume dist-client-22 >244: type protocol/client >245: option ping-timeout 42 >246: option remote-host 10.70.35.198 >247: option remote-subvolume /brick2/dist24 >248: option transport-type socket >249: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >250: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >251: option send-gids true >252: end-volume >253: >254: volume dist-client-23 >255: type protocol/client >256: option ping-timeout 42 >257: option remote-host 10.70.35.240 >258: option remote-subvolume /brick2/dist23 >259: option transport-type socket >260: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >261: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >262: option send-gids true >263: end-volume >264: >265: volume dist-client-24 >266: type protocol/client >267: option ping-timeout 42 >268: option remote-host 10.70.35.172 >269: option remote-subvolume /brick2/dist22 >270: option transport-type socket >271: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >272: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >273: option send-gids true >274: end-volume >275: >276: volume dist-client-25 >277: type protocol/client >278: option ping-timeout 42 >279: option remote-host 10.70.35.198 >280: option remote-subvolume /brick2/dist27 >281: option transport-type socket >282: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >283: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >284: option send-gids true >285: end-volume >286: >287: volume dist-client-26 >288: type protocol/client >289: option ping-timeout 42 >290: option remote-host 10.70.35.240 >291: option remote-subvolume /brick2/dist26 >292: option transport-type socket >293: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >294: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >295: option send-gids true >296: end-volume >297: >298: volume dist-client-27 >299: type protocol/client >300: option ping-timeout 42 >301: option remote-host 10.70.35.172 >302: option remote-subvolume /brick2/dist25 >303: option transport-type socket >304: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >305: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >306: option send-gids true >307: end-volume >308: >309: volume dist-client-28 >310: type protocol/client >311: option ping-timeout 42 >312: option remote-host 10.70.35.198 >313: option remote-subvolume /brick2/dist30 >314: option transport-type socket >315: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >316: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >317: option send-gids true >318: end-volume >319: >320: volume dist-client-29 >321: type protocol/client >322: option ping-timeout 42 >323: option remote-host 10.70.35.240 >324: option remote-subvolume /brick2/dist29 >325: option transport-type socket >326: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >327: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >328: option send-gids true >329: end-volume >330: >331: volume dist-client-30 >332: type protocol/client >333: option ping-timeout 42 >334: option remote-host 10.70.35.172 >335: option remote-subvolume /brick2/dist28 >336: option transport-type socket >337: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >338: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >339: option send-gids true >340: end-volume >341: >342: volume dist-client-31 >343: type protocol/client >344: option ping-timeout 42 >345: option remote-host 10.70.35.198 >346: option remote-subvolume /brick2/dist33 >347: option transport-type socket >348: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >349: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >350: option send-gids true >351: end-volume >352: >353: volume dist-client-32 >354: type protocol/client >355: option ping-timeout 42 >356: option remote-host 10.70.35.240 >357: option remote-subvolume /brick2/dist32 >358: option transport-type socket >359: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >360: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >361: option send-gids true >362: end-volume >363: >364: volume dist-client-33 >365: type protocol/client >366: option ping-timeout 42 >367: option remote-host 10.70.35.172 >368: option remote-subvolume /brick2/dist31 >369: option transport-type socket >370: option username cfbd03d8-7075-45ed-a959-ac868dd44f85 >371: option password 7f49e84f-19e8-453f-9352-7a9e5c58e810 >372: option send-gids true >373: end-volume >374: >375: volume dist-dht >376: type cluster/distribute >377: subvolumes dist-client-0 dist-client-1 dist-client-2 dist-client-3 dist-client-4 dist-client-5 dist-client-6 dist-client-7 dist-client-8 dist-client-9 dist-client-10 dist-client-11 dist-client-12 dist-client-13 dist-client-14 dist-client-15 dist-client-16 dist-client-17 dist-client-18 dist-client-19 dist-client-20 dist-client-21 dist-client-22 dist-client-23 dist-client-24 dist-client-25 dist-client-26 dist-client-27 dist-client-28 dist-client-29 dist-client-30 dist-client-31 dist-client-32 dist-client-33 >378: end-volume >379: >380: volume dist-write-behind >381: type performance/write-behind >382: subvolumes dist-dht >383: end-volume >384: >385: volume dist >386: type debug/io-stats >387: option latency-measurement off >388: option count-fop-hits off >389: subvolumes dist-write-behind >390: end-volume >391: >392: volume test0-client-0 >393: type protocol/client >394: option ping-timeout 42 >395: option remote-host 10.70.35.198 >396: option remote-subvolume /brick0 >397: option transport-type socket >398: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >399: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >400: option send-gids true >401: end-volume >402: >403: volume test0-client-1 >404: type protocol/client >405: option ping-timeout 42 >406: option remote-host 10.70.35.240 >407: option remote-subvolume /brick0 >408: option transport-type socket >409: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >410: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >411: option send-gids true >412: end-volume >413: >414: volume test0-client-2 >415: type protocol/client >416: option ping-timeout 42 >417: option remote-host 10.70.35.172 >418: option remote-subvolume /brick0 >419: option transport-type socket >420: option username e1d9b6f0-5bf4-4228-835b-ea66fa772558 >421: option password 49eb411d-3c6e-4a82-b2d2-fd8b6b7087dd >422: option send-gids true >423: end-volume >424: >425: volume test0-dht >426: type cluster/distribute >427: subvolumes test0-client-0 test0-client-1 test0-client-2 >428: end-volume >429: >430: volume test0-write-behind >431: type performance/write-behind >432: subvolumes test0-dht >433: end-volume >434: >435: volume test0 >436: type debug/io-stats >437: option latency-measurement off >438: option count-fop-hits off >439: subvolumes test0-write-behind >440: end-volume >441: >442: volume test1-client-0 >443: type protocol/client >444: option ping-timeout 42 >445: option remote-host 10.70.35.198 >446: option remote-subvolume /brick/1 >447: option transport-type socket >448: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >449: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >450: option send-gids true >451: end-volume >452: >453: volume test1-client-1 >454: type protocol/client >455: option ping-timeout 42 >456: option remote-host 10.70.35.198 >457: option remote-subvolume /brick/2 >458: option transport-type socket >459: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >460: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >461: option send-gids true >462: end-volume >463: >464: volume test1-client-2 >465: type protocol/client >466: option ping-timeout 42 >467: option remote-host 10.70.35.198 >468: option remote-subvolume /brick/3 >469: option transport-type socket >470: option username 012e81fa-c9ea-4ef9-a8bf-6bb27f9cb622 >471: option password 25dbc1ee-f2b3-409d-a3a6-c8268a8331fb >472: option send-gids true >473: end-volume >474: >475: volume test1-dht >476: type cluster/distribute >477: subvolumes test1-client-0 test1-client-1 test1-client-2 >478: end-volume >479: >480: volume test1-write-behind >481: type performance/write-behind >482: subvolumes test1-dht >483: end-volume >484: >485: volume test1 >486: type debug/io-stats >487: option latency-measurement off >488: option count-fop-hits off >489: subvolumes test1-write-behind >490: end-volume >491: >492: volume workstation-client-0 >493: type protocol/client >494: option ping-timeout 42 >495: option remote-host 10.70.35.198 >496: option remote-subvolume /brick2/workstation0 >497: option transport-type socket >498: option username a7fd1896-75f7-4ea6-afcf-9eca43de8d10 >499: option password 78bbb91d-e14d-46c5-a3d1-f6a97d973c38 >500: option send-gids true >501: end-volume >502: >503: volume workstation-client-1 >504: type protocol/client >505: option ping-timeout 42 >506: option remote-host 10.70.35.240 >507: option remote-subvolume /brick2/workstation1 >508: option transport-type socket >509: option username a7fd1896-75f7-4ea6-afcf-9eca43de8d10 >510: option password 78bbb91d-e14d-46c5-a3d1-f6a97d973c38 >511: option send-gids true >512: end-volume >513: >514: volume workstation-client-2 >515: type protocol/client >516: option ping-timeout 42 >517: option remote-host 10.70.35.172 >518: option remote-subvolume /brick2/workstation2 >519: option transport-type socket >520: option username a7fd1896-75f7-4ea6-afcf-9eca43de8d10 >521: option password 78bbb91d-e14d-46c5-a3d1-f6a97d973c38 >522: option send-gids true >523: end-volume >524: >525: volume workstation-dht >526: type cluster/distribute >527: subvolumes workstation-client-0 workstation-client-1 workstation-client-2 >528: end-volume >529: >530: volume workstation-write-behind >531: type performance/write-behind >532: subvolumes workstation-dht >533: end-volume >534: >535: volume workstation >536: type debug/io-stats >537: option latency-measurement off >538: option count-fop-hits off >539: subvolumes workstation-write-behind >540: end-volume >541: >542: volume nfs-server >543: type nfs/server >544: option rpc-auth.auth-glusterfs on >545: option rpc-auth.auth-unix on >546: option rpc-auth.auth-null on >547: option rpc-auth.ports.insecure on >548: option rpc-auth-allow-insecure on >549: option transport-type socket >550: option transport.socket.listen-port 2049 >551: option nfs.dynamic-volumes on >552: option nfs.nlm on >553: option nfs.drc off >554: option rpc-auth.addr.dist.allow * >555: option nfs3.dist.volume-id f698a6fa-51cc-4e39-90e8-7050fe47d51f >556: option rpc-auth.addr.test0.allow * >557: option nfs3.test0.volume-id 0ecb97da-2c48-46b7-896f-6ba851b84d2f >558: option rpc-auth.addr.test1.allow * >559: option nfs3.test1.volume-id 2b60f468-d6ea-48b1-851e-c11f4c4e41a7 >560: option rpc-auth.addr.workstation.allow * >561: option nfs3.workstation.volume-id e5c0899d-d49a-485a-a821-23db66705470 >562: subvolumes dist test0 test1 workstation >563: end-volume >564: >+------------------------------------------------------------------------------+ >[2014-08-21 10:25:12.777230] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-0: changing port to 49192 (from 0) >[2014-08-21 10:25:12.777305] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-3: changing port to 49193 (from 0) >[2014-08-21 10:25:12.777366] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-4: changing port to 49194 (from 0) >[2014-08-21 10:25:12.777422] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-5: changing port to 49195 (from 0) >[2014-08-21 10:25:12.777527] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-6: changing port to 49196 (from 0) >[2014-08-21 10:25:12.777586] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-7: changing port to 49197 (from 0) >[2014-08-21 10:25:12.777676] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-8: changing port to 49198 (from 0) >[2014-08-21 10:25:12.777735] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-10: changing port to 49199 (from 0) >[2014-08-21 10:25:12.777825] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-13: changing port to 49200 (from 0) >[2014-08-21 10:25:12.777884] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-16: changing port to 49201 (from 0) >[2014-08-21 10:25:12.777973] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-19: changing port to 49202 (from 0) >[2014-08-21 10:25:12.778032] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-22: changing port to 49203 (from 0) >[2014-08-21 10:25:12.778150] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-25: changing port to 49204 (from 0) >[2014-08-21 10:25:12.778211] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-28: changing port to 49205 (from 0) >[2014-08-21 10:25:12.778298] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-31: changing port to 49206 (from 0) >[2014-08-21 10:25:12.778361] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-0: changing port to 49187 (from 0) >[2014-08-21 10:25:12.778419] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-0: changing port to 49189 (from 0) >[2014-08-21 10:25:12.778539] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-1: changing port to 49190 (from 0) >[2014-08-21 10:25:12.778599] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test1-client-2: changing port to 49191 (from 0) >[2014-08-21 10:25:12.861508] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-workstation-client-0: changing port to 49208 (from 0) >[2014-08-21 10:25:12.867635] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.867767] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.867898] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.867996] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.868362] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.868481] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.868722] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.868825] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.868928] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-13: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869098] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-16: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869235] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-19: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869370] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-22: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869488] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-25: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869626] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-28: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869732] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-31: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.869864] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.870199] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.870307] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.870738] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-0: Connected to dist-client-0, attached to remote volume '/brick2/dist0'. >[2014-08-21 10:25:12.870760] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.870938] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-3: Connected to dist-client-3, attached to remote volume '/brick2/dist3'. >[2014-08-21 10:25:12.870967] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-3: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.871170] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-4: Connected to dist-client-4, attached to remote volume '/brick2/dist4'. >[2014-08-21 10:25:12.871202] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-4: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.871362] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-5: Connected to dist-client-5, attached to remote volume '/brick2/dist5'. >[2014-08-21 10:25:12.871390] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-5: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.871773] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-6: Connected to dist-client-6, attached to remote volume '/brick2/dist6'. >[2014-08-21 10:25:12.871795] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-6: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.871894] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-7: Connected to dist-client-7, attached to remote volume '/brick2/dist7'. >[2014-08-21 10:25:12.871913] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-7: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.872296] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-8: Connected to dist-client-8, attached to remote volume '/brick2/dist8'. >[2014-08-21 10:25:12.872320] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-8: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.872442] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-10: Connected to dist-client-10, attached to remote volume '/brick2/dist10'. >[2014-08-21 10:25:12.872476] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-10: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.872627] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-13: Connected to dist-client-13, attached to remote volume '/brick2/dist13'. >[2014-08-21 10:25:12.872657] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-13: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.872816] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-16: Connected to dist-client-16, attached to remote volume '/brick2/dist16'. >[2014-08-21 10:25:12.872846] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-16: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.873007] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-19: Connected to dist-client-19, attached to remote volume '/brick2/dist19'. >[2014-08-21 10:25:12.873035] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-19: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.873259] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-22: Connected to dist-client-22, attached to remote volume '/brick2/dist24'. >[2014-08-21 10:25:12.873282] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-22: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.873431] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-25: Connected to dist-client-25, attached to remote volume '/brick2/dist27'. >[2014-08-21 10:25:12.873476] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-25: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.873664] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-28: Connected to dist-client-28, attached to remote volume '/brick2/dist30'. >[2014-08-21 10:25:12.873696] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-28: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.873856] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-31: Connected to dist-client-31, attached to remote volume '/brick2/dist33'. >[2014-08-21 10:25:12.873895] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-31: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.874119] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-0: Connected to test0-client-0, attached to remote volume '/brick0'. >[2014-08-21 10:25:12.874143] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.874541] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-0: Connected to test1-client-0, attached to remote volume '/brick/1'. >[2014-08-21 10:25:12.874564] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.874718] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-2: Connected to test1-client-2, attached to remote volume '/brick/3'. >[2014-08-21 10:25:12.874747] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.875162] I [client-handshake.c:1415:select_server_supported_programs] 0-test1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.875268] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-0: Server lk version = 1 >[2014-08-21 10:25:12.875296] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-3: Server lk version = 1 >[2014-08-21 10:25:12.875316] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-4: Server lk version = 1 >[2014-08-21 10:25:12.875337] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-5: Server lk version = 1 >[2014-08-21 10:25:12.875356] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-6: Server lk version = 1 >[2014-08-21 10:25:12.875376] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-7: Server lk version = 1 >[2014-08-21 10:25:12.875395] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-8: Server lk version = 1 >[2014-08-21 10:25:12.875415] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-10: Server lk version = 1 >[2014-08-21 10:25:12.875434] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-13: Server lk version = 1 >[2014-08-21 10:25:12.875469] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-16: Server lk version = 1 >[2014-08-21 10:25:12.875491] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-19: Server lk version = 1 >[2014-08-21 10:25:12.875511] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-22: Server lk version = 1 >[2014-08-21 10:25:12.875531] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-25: Server lk version = 1 >[2014-08-21 10:25:12.875550] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-28: Server lk version = 1 >[2014-08-21 10:25:12.875569] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-31: Server lk version = 1 >[2014-08-21 10:25:12.875589] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-0: Server lk version = 1 >[2014-08-21 10:25:12.875609] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-0: Server lk version = 1 >[2014-08-21 10:25:12.875628] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-2: Server lk version = 1 >[2014-08-21 10:25:12.875651] I [client-handshake.c:1415:select_server_supported_programs] 0-workstation-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:12.875878] I [client-handshake.c:1200:client_setvolume_cbk] 0-test1-client-1: Connected to test1-client-1, attached to remote volume '/brick/2'. >[2014-08-21 10:25:12.875898] I [client-handshake.c:1212:client_setvolume_cbk] 0-test1-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.876551] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test1-client-1: Server lk version = 1 >[2014-08-21 10:25:12.876601] I [client-handshake.c:1200:client_setvolume_cbk] 0-workstation-client-0: Connected to workstation-client-0, attached to remote volume '/brick2/workstation0'. >[2014-08-21 10:25:12.876617] I [client-handshake.c:1212:client_setvolume_cbk] 0-workstation-client-0: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:12.877192] I [client-handshake.c:188:client_set_lk_version_cbk] 0-workstation-client-0: Server lk version = 1 >[2014-08-21 10:25:13.552825] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-1: changing port to 49175 (from 0) >[2014-08-21 10:25:13.557585] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-9: changing port to 49176 (from 0) >[2014-08-21 10:25:13.557640] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-11: changing port to 49177 (from 0) >[2014-08-21 10:25:13.557685] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-14: changing port to 49178 (from 0) >[2014-08-21 10:25:13.557826] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-17: changing port to 49179 (from 0) >[2014-08-21 10:25:13.558253] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-20: changing port to 49180 (from 0) >[2014-08-21 10:25:13.580384] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-23: changing port to 49181 (from 0) >[2014-08-21 10:25:13.580441] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-26: changing port to 49182 (from 0) >[2014-08-21 10:25:13.580487] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-29: changing port to 49183 (from 0) >[2014-08-21 10:25:13.580519] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-32: changing port to 49184 (from 0) >[2014-08-21 10:25:13.580561] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-1: changing port to 49173 (from 0) >[2014-08-21 10:25:13.580605] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-workstation-client-1: changing port to 49185 (from 0) >[2014-08-21 10:25:13.608633] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.608761] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.608880] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.608981] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-14: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.609144] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-2: changing port to 49177 (from 0) >[2014-08-21 10:25:13.609196] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-12: changing port to 49178 (from 0) >[2014-08-21 10:25:13.609283] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-15: changing port to 49179 (from 0) >[2014-08-21 10:25:13.609360] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-18: changing port to 49180 (from 0) >[2014-08-21 10:25:13.609432] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-21: changing port to 49181 (from 0) >[2014-08-21 10:25:13.609470] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-24: changing port to 49182 (from 0) >[2014-08-21 10:25:13.609552] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-27: changing port to 49183 (from 0) >[2014-08-21 10:25:13.609659] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-30: changing port to 49184 (from 0) >[2014-08-21 10:25:13.609754] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-dist-client-33: changing port to 49185 (from 0) >[2014-08-21 10:25:13.609838] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-test0-client-2: changing port to 49175 (from 0) >[2014-08-21 10:25:13.609887] I [rpc-clnt.c:1759:rpc_clnt_reconfig] 0-workstation-client-2: changing port to 49186 (from 0) >[2014-08-21 10:25:13.610247] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-11: Connected to dist-client-11, attached to remote volume '/brick2/dist11'. >[2014-08-21 10:25:13.610330] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-11: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.610523] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-9: Connected to dist-client-9, attached to remote volume '/brick2/dist9'. >[2014-08-21 10:25:13.610544] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-9: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.610711] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-14: Connected to dist-client-14, attached to remote volume '/brick2/dist14'. >[2014-08-21 10:25:13.610731] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-14: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.610882] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-23: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.659148] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-20: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.659234] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-17: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.659393] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-1: Connected to dist-client-1, attached to remote volume '/brick2/dist1'. >[2014-08-21 10:25:13.659415] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.659552] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-11: Server lk version = 1 >[2014-08-21 10:25:13.659581] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-9: Server lk version = 1 >[2014-08-21 10:25:13.659601] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-14: Server lk version = 1 >[2014-08-21 10:25:13.659629] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-23: Connected to dist-client-23, attached to remote volume '/brick2/dist23'. >[2014-08-21 10:25:13.659643] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-23: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.659767] I [client-handshake.c:1415:select_server_supported_programs] 0-workstation-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.659890] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-26: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.659962] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.660093] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-29: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.660192] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-32: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.661108] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-23: Server lk version = 1 >[2014-08-21 10:25:13.661192] I [client-handshake.c:1200:client_setvolume_cbk] 0-workstation-client-1: Connected to workstation-client-1, attached to remote volume '/brick2/workstation1'. >[2014-08-21 10:25:13.661212] I [client-handshake.c:1212:client_setvolume_cbk] 0-workstation-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.661411] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-26: Connected to dist-client-26, attached to remote volume '/brick2/dist26'. >[2014-08-21 10:25:13.661432] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-26: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.661570] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-1: Connected to test0-client-1, attached to remote volume '/brick0'. >[2014-08-21 10:25:13.661598] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-1: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.661751] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-29: Connected to dist-client-29, attached to remote volume '/brick2/dist29'. >[2014-08-21 10:25:13.661792] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-29: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.661946] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-32: Connected to dist-client-32, attached to remote volume '/brick2/dist32'. >[2014-08-21 10:25:13.661966] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-32: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.662110] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662252] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-12: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662369] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-15: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662472] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-18: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662549] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-21: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662644] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-24: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662719] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-27: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662818] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-30: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.662928] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-20: Connected to dist-client-20, attached to remote volume '/brick2/dist20'. >[2014-08-21 10:25:13.662948] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-20: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.663096] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-1: Server lk version = 1 >[2014-08-21 10:25:13.663131] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-17: Connected to dist-client-17, attached to remote volume '/brick2/dist17'. >[2014-08-21 10:25:13.663172] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-17: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.663432] I [client-handshake.c:188:client_set_lk_version_cbk] 0-workstation-client-1: Server lk version = 1 >[2014-08-21 10:25:13.663486] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-26: Server lk version = 1 >[2014-08-21 10:25:13.663545] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-1: Server lk version = 1 >[2014-08-21 10:25:13.663570] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-29: Server lk version = 1 >[2014-08-21 10:25:13.663590] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-32: Server lk version = 1 >[2014-08-21 10:25:13.663617] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-2: Connected to dist-client-2, attached to remote volume '/brick2/dist2'. >[2014-08-21 10:25:13.663632] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.663767] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-12: Connected to dist-client-12, attached to remote volume '/brick2/dist12'. >[2014-08-21 10:25:13.663794] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-12: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.663905] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-15: Connected to dist-client-15, attached to remote volume '/brick2/dist15'. >[2014-08-21 10:25:13.663923] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-15: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.664081] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-18: Connected to dist-client-18, attached to remote volume '/brick2/dist18'. >[2014-08-21 10:25:13.664101] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-18: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.664254] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-21: Connected to dist-client-21, attached to remote volume '/brick2/dist21'. >[2014-08-21 10:25:13.664352] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-21: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.664472] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-24: Connected to dist-client-24, attached to remote volume '/brick2/dist22'. >[2014-08-21 10:25:13.664513] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-24: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.664727] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-27: Connected to dist-client-27, attached to remote volume '/brick2/dist25'. >[2014-08-21 10:25:13.664777] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-27: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.664900] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-20: Server lk version = 1 >[2014-08-21 10:25:13.664961] I [client-handshake.c:1415:select_server_supported_programs] 0-dist-client-33: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.665087] I [client-handshake.c:1415:select_server_supported_programs] 0-test0-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.665169] I [client-handshake.c:1415:select_server_supported_programs] 0-workstation-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) >[2014-08-21 10:25:13.665407] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-2: Server lk version = 1 >[2014-08-21 10:25:13.665437] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-12: Server lk version = 1 >[2014-08-21 10:25:13.665458] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-15: Server lk version = 1 >[2014-08-21 10:25:13.665537] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-18: Server lk version = 1 >[2014-08-21 10:25:13.665579] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-21: Server lk version = 1 >[2014-08-21 10:25:13.665645] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-24: Server lk version = 1 >[2014-08-21 10:25:13.665685] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-27: Server lk version = 1 >[2014-08-21 10:25:13.665737] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-30: Connected to dist-client-30, attached to remote volume '/brick2/dist28'. >[2014-08-21 10:25:13.665752] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-30: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.665864] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-17: Server lk version = 1 >[2014-08-21 10:25:13.666163] I [client-handshake.c:1200:client_setvolume_cbk] 0-dist-client-33: Connected to dist-client-33, attached to remote volume '/brick2/dist31'. >[2014-08-21 10:25:13.666203] I [client-handshake.c:1212:client_setvolume_cbk] 0-dist-client-33: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.666764] I [client-handshake.c:1200:client_setvolume_cbk] 0-test0-client-2: Connected to test0-client-2, attached to remote volume '/brick0'. >[2014-08-21 10:25:13.666788] I [client-handshake.c:1212:client_setvolume_cbk] 0-test0-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.667403] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-33: Server lk version = 1 >[2014-08-21 10:25:13.667437] I [client-handshake.c:188:client_set_lk_version_cbk] 0-test0-client-2: Server lk version = 1 >[2014-08-21 10:25:13.667458] I [client-handshake.c:188:client_set_lk_version_cbk] 0-dist-client-30: Server lk version = 1 >[2014-08-21 10:25:13.671091] I [client-handshake.c:1200:client_setvolume_cbk] 0-workstation-client-2: Connected to workstation-client-2, attached to remote volume '/brick2/workstation2'. >[2014-08-21 10:25:13.671127] I [client-handshake.c:1212:client_setvolume_cbk] 0-workstation-client-2: Server and Client lk-version numbers are not same, reopening the fds >[2014-08-21 10:25:13.671864] I [client-handshake.c:188:client_set_lk_version_cbk] 0-workstation-client-2: Server lk version = 1 >[2014-08-21 10:25:17.482326] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-workstation-client-2: remote operation failed: Stale file handle >[2014-08-21 10:25:17.484989] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-workstation-client-2: remote operation failed: No such file or directory. Path: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> (317a9b66-0901-41fb-a9f0-0960ec7d62bf) >[2014-08-21 10:25:17.485179] E [dht-helper.c:813:dht_migration_complete_check_task] 0-workstation-dht: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf>: failed to lookup the file on workstation-client-2 >[2014-08-21 10:25:17.485238] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 3a2e8783: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> => -1 (No such file or directory) >[2014-08-21 10:25:17.485260] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 3a2e8783, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 10:25:17.486675] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-workstation-client-2: remote operation failed: Stale file handle >[2014-08-21 10:25:17.489027] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-workstation-client-2: remote operation failed: No such file or directory. Path: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> (317a9b66-0901-41fb-a9f0-0960ec7d62bf) >[2014-08-21 10:25:17.489206] E [dht-helper.c:813:dht_migration_complete_check_task] 0-workstation-dht: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf>: failed to lookup the file on workstation-client-2 >[2014-08-21 10:25:17.489242] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 3b2e8783: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> => -1 (No such file or directory) >[2014-08-21 10:25:17.489260] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 3b2e8783, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 10:25:17.490582] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-workstation-client-2: remote operation failed: Stale file handle >[2014-08-21 10:25:17.494642] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-workstation-client-2: remote operation failed: No such file or directory. Path: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> (317a9b66-0901-41fb-a9f0-0960ec7d62bf) >[2014-08-21 10:25:17.494827] E [dht-helper.c:813:dht_migration_complete_check_task] 0-workstation-dht: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf>: failed to lookup the file on workstation-client-2 >[2014-08-21 10:25:17.494862] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 3c2e8783: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> => -1 (No such file or directory) >[2014-08-21 10:25:17.494880] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 3c2e8783, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory) >[2014-08-21 10:25:17.496154] W [client-rpc-fops.c:1357:client3_3_access_cbk] 0-workstation-client-2: remote operation failed: Stale file handle >[2014-08-21 10:25:17.498913] W [client-rpc-fops.c:2761:client3_3_lookup_cbk] 0-workstation-client-2: remote operation failed: No such file or directory. Path: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> (317a9b66-0901-41fb-a9f0-0960ec7d62bf) >[2014-08-21 10:25:17.499063] E [dht-helper.c:813:dht_migration_complete_check_task] 0-workstation-dht: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf>: failed to lookup the file on workstation-client-2 >[2014-08-21 10:25:17.499097] W [nfs3.c:1532:nfs3svc_access_cbk] 0-nfs: 3d2e8783: <gfid:317a9b66-0901-41fb-a9f0-0960ec7d62bf> => -1 (No such file or directory) >[2014-08-21 10:25:17.499118] W [nfs3-helpers.c:3401:nfs3_log_common_res] 0-nfs-nfsv3: XID: 3d2e8783, ACCESS: NFS: 2(No such file or directory), POSIX: 2(No such file or directory)
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1132489
: 929196